How to use Codex to check patches for regressions
Trail of Bits used agent checks and before-and-after proofs to review merged security patches, confirming 21 regressions. Its regression-hunt plugin adds independent replay and evidence adjudication.
blog.trailofbits.com
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Review [patch or pull request] in [repository] for regressions. Trace changed code through callers to supported user interfaces. Run each candidate's proof of concept against [before revision] and [after revision] on live instances. Record commands, raw evidence, and open questions. Treat candidates without a live proof as inconclusive, and prepare findings for human review.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source poster / publisher
- tob_scott_a
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The review confirmed 21 regressions across more than 130 merged pull requests, with none breaking a security protection. Preliminary evaluations reported 25% fewer false positives for regression-hunt than for a baseline Codex agent prompted to review a PR for regressions.
- Human involvement
- Engineers reviewed agent results, and the team confirmed which reported candidates were regressions.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Supplied merged patches for agent review of potential regressions.
- Had an agent run proofs of concept before and after each change.
- Passed the agent's findings and evidence to an engineer for review.
Trail of Bits used agents to review security patches developed through Patch the Planet, its joint initiative with OpenAI. After earlier patches introduced regressions, the team reviewed every patch that a Trail of Bits engineer had authored and maintainers had merged upstream. Its plugin design also drew on thousands of engineers’ Codex sessions.
For each patch, an agent searched for potential regressions and tested a proof of concept before and after the change. It then passed the results to an engineer for review. The review confirmed 21 regressions across more than 130 merged pull requests: 16 functional, one performance, and four build or test regressions. None broke a security protection. A single run produced 58 candidates, illustrating the manual sorting required.
The team published regression-hunt to standardize this work. Its workflow traces changed code through callers to real interfaces, reproduces candidates on a live deployment, has a fresh-context agent replay the proof, and judges whether the evidence establishes a regression. Conclusions without a live proof remain inconclusive. Preliminary evaluations reported 25% fewer false positives than a baseline Codex review prompt.
Who it's for: Engineers and maintainers reviewing patches that could disrupt supported behavior.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Blog · tob_scott_a
Frequently asked questions
How were suspected regressions checked?
An agent ran a proof of concept before and after each change, then passed its results to an engineer for review. The published plugin also includes replay by a fresh-context agent.
What did the review actually find?
The review confirmed 21 regressions across more than 130 merged pull requests: 16 functional, one performance, and four build or test regressions. None broke a security protection.
What happens when a live proof cannot be produced?
The article says a step is inconclusive if it cannot produce a live proof of concept when reproducing a bug, proving a fix, or confirming a regression.
More details
Remaining steps
- Used the patch review and Codex sessions to design standardized plugin workflows.
About this account
Agents investigated possible regressions and ran before-and-after proofs; engineers reviewed the resulting findings.
Platform: Blog. Attribution: https://blog.trailofbits.com/2026/10/21/your-ai-agent-fixed-the-bug-but-what-else-did-it-change/. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: tob_scott_a on Blog
Source published
Added to Agents In Use
Source checked
Tools and topics
Codex · regression-hunt
Regression testing · Security patches · Code review
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.