How to use Claude to fix flaky CI tests
An engineering leader gave Claude CI artifact access and pass-rate tooling, let it investigate and rerun flaky tests overnight, and reviewed its proposed fixes during the day.
news.ycombinator.com
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Using read access to [CI artifacts], build a rolling pass/fail report for [time window]. Rank flaky tests, investigate test versus production causes, and seek minimal deterministic reproductions. Propose fixes with [validation runs] and evidence of remaining uncertainty, then prepare pull requests for my review.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- SeanAnderson
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The leader reports removing 20 minutes from the CI pipeline and improving pass rate by 35% in a handful of weeks; the source does not specify how that percentage was calculated.
- Human involvement
- The leader reviewed PR proposals between meetings and explicitly discarded some bad solutions.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Built tooling to monitor 30 days of rolling CI pass and fail rates.
- Identified the worst flaky tests and formed failure hypotheses.
- Used divide-and-conquer investigation to seek minimal reproductions.
An engineering leader described using Claude to rehabilitate a legacy team's unreliable CI pipeline. The suite took roughly 40 minutes and the main branch passed about 40% of the time despite repeated test reruns. The leader gave Claude read access to CI artifacts and had it build tools to monitor the rolling pass/fail rate over the previous 30 days.
Claude identified the worst flaky tests, formed hypotheses about whether failures originated in tests or production code, and used divide-and-conquer investigation to seek minimal reproductions. When it could not produce a deterministic reproduction, it made a proposed fix and repeatedly reran tests overnight to assess whether the change appeared successful with statistical confidence.
The leader reviewed proposed pull requests during daytime gaps between meetings and discarded some bad solutions. After a handful of weeks, they reported removing 20 minutes from the pipeline and improving its pass rate by 35%. The source does not clarify whether that improvement means a relative percentage or percentage points. It also does not describe the statistical method used. Who it's for: engineering teams with accessible CI artifacts and capacity to review agent-generated fixes to a flaky test suite.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Hacker News · SeanAnderson
Frequently asked questions
What information did Claude receive before investigating?
The leader gave Claude read access to CI artifacts and had it build tooling to monitor the rolling pass/fail rate over the previous 30 days.
How did the leader review Claude’s work?
Claude ran overnight, while the leader reviewed proposed pull requests during the day between meetings and discarded some bad solutions.
What improvement did the leader report?
They reported shaving 20 minutes off the pipeline and improving pass rate by 35% over a handful of weeks. The source does not distinguish relative percentage from percentage points.
More details
Remaining steps
- Proposed fixes and reran tests overnight when deterministic reproduction failed.
- Reviewed proposed pull requests and discarded bad solutions.
About this account
Claude investigated failures, proposed fixes and reran tests overnight while the leader reviewed proposals during the day.
Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=47970174. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: SeanAnderson on Hacker News
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude · CI artifacts · Pass-rate monitoring tools
CI · Flaky tests · Legacy software
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.