How to use Claude to fix flaky CI tests
An engineering leader gave Claude CI artifact access and pass-rate tooling, let it investigate and rerun flaky tests overnight, and reviewed its proposed fixes during the day.
news.ycombinator.com
Prompt to try
Editorial adaptation, not the source author’s exact prompt. Replace bracketed inputs and review tool permissions first.
Using read access to [CI artifacts], build a rolling pass/fail report for [time window]. Rank flaky tests, investigate test versus production causes, and seek minimal deterministic reproductions. Propose fixes with [validation runs] and evidence of remaining uncertainty, then prepare pull requests for my review.
Reach people exploring AI agents
See placements →What happened
The leader reports removing 20 minutes from the CI pipeline and improving pass rate by 35% in a handful of weeks; the source does not specify how that percentage was calculated.
Reported by SeanAnderson on Hacker NewsHow the agent did it
- Built tooling to monitor 30 days of rolling CI pass and fail rates.
- Identified the worst flaky tests and formed failure hypotheses.
- Used divide-and-conquer investigation to seek minimal reproductions.
Where a person stepped in
The leader reviewed PR proposals between meetings and explicitly discarded some bad solutions.
A person approved each step before the agent continued.
An engineering leader described using Claude to rehabilitate a legacy team's unreliable CI pipeline. The suite took roughly 40 minutes and the main branch passed about 40% of the time despite repeated test reruns. The leader gave Claude read access to CI artifacts and had it build tools to monitor the rolling pass/fail rate over the previous 30 days.
Claude identified the worst flaky tests, formed hypotheses about whether failures originated in tests or production code, and used divide-and-conquer investigation to seek minimal reproductions. When it could not produce a deterministic reproduction, it made a proposed fix and repeatedly reran tests overnight to assess whether the change appeared successful with statistical confidence.
The leader reviewed proposed pull requests during daytime gaps between meetings and discarded some bad solutions. After a handful of weeks, they reported removing 20 minutes from the pipeline and improving its pass rate by 35%. The source does not clarify whether that improvement means a relative percentage or percentage points. It also does not describe the statistical method used. Who it's for: engineering teams with accessible CI artifacts and capacity to review agent-generated fixes to a flaky test suite.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Hacker News · SeanAnderson
Frequently asked questions
What information did Claude receive before investigating?
The leader gave Claude read access to CI artifacts and had it build tooling to monitor the rolling pass/fail rate over the previous 30 days.
How did the leader review Claude’s work?
Claude ran overnight, while the leader reviewed proposed pull requests during the day between meetings and discarded some bad solutions.
What improvement did the leader report?
They reported shaving 20 minutes off the pipeline and improving pass rate by 35% over a handful of weeks. The source does not distinguish relative percentage from percentage points.
More details
Remaining steps
- Proposed fixes and reran tests overnight when deterministic reproduction failed.
- Reviewed proposed pull requests and discarded bad solutions.
About this account
Claude investigated failures, proposed fixes and reran tests overnight while the leader reviewed proposals during the day.
Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=47970174. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: SeanAnderson on Hacker News
Source published 1 May 2026
Source checked 14 Oct 2026
Tools and topics
Claude · CI artifacts · Pass-rate monitoring tools
CI · Flaky tests · Legacy software
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.