How to use Codex to test CVE patches independently
Codex and Claude Code attempted isolated CVE patches. Attack proofs rejected fixes that passed normal tests; one strongly guided Codex attempt succeeded.
kkm-mako.com
Original prompt
Shared by the source author. See source notes for context.
Fix the one issue you judge most serious, with the minimal necessary change.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- Makoto Horikawa
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- One of seventeen reported attempts stopped its attack, a strongest-hint Codex patch. Existing tests passed even for ineffective fixes; calibrated external proofs supplied the decisive check.
- Human involvement
- Horikawa isolated repositories, varied hints, repaired faulty grading scripts and judged patches with a separately held attack reproduction.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Horikawa gave isolated vulnerable repository copies to Codex and Claude Code with varied hints.
- He repaired and calibrated independent proof-of-concept graders against known vulnerable and fixed versions.
- Only the strongest-hint Codex patch stopped its attack in the reported trials.
Makoto Horikawa compared Codex and Claude Code on patches for vm2 and axios vulnerabilities using isolated repository copies. Internet access, useful history and answer-bearing metadata were removed so the agents had to work from the supplied code rather than retrieve an upstream correction.
The experiment varied hints from a broad security request to a detailed description of the broken invariant. Its decisive check was a separately held proof of concept: the attack had to stop. Passing the library's own tests and writing persuasive fix notes were supplementary evidence, not the result.
The grader itself needed repair. One reproduction depended on a file absent from the vulnerable checkout; another read the wrong build artifact. Horikawa fixed these problems and confirmed that the grader distinguished known vulnerable and fixed versions before accepting agent results.
Only one of the reported seventeen attempts blocked its attack, a Codex run with the strongest hint. That patch wrapped a host-derived value instead of letting it cross the sandbox boundary raw. The account is useful as a concrete patch-verification experiment, including failed fixes that passed existing tests. It does not establish general superiority across models or projects, and the patches were experimental rather than reported upstream merges.
Prompt
Original prompt
Source: Research blog · Makoto Horikawa
Frequently asked questions
What decided whether a patch passed?
A separate proof of concept had to stop exploiting the issue.
Why repair the grader first?
Missing files and wrong build artifacts initially produced misleading judgments.
Can this rank all coding agents?
No. It is a limited seventeen-trial experiment on selected vulnerabilities.
More details
About this account
Horikawa gave isolated vulnerable repository copies to Codex and Claude Code with varied hints. He repaired and calibrated independent proof-of-concept graders against known vulnerable and fixed versions.
Original English article and Japanese original HTML read. Prompt excerpt is the author's published English translation; actual runs used Japanese. Seventeen trials are a limited experiment, not a general model ranking or upstream fix claim. Original prompt excerpt.
Credit: Makoto Horikawa — Research blog
Source published
Added to Agents In Use
Source checked
Tools and topics
Codex · Claude Code · bubblewrap · vm2 · axios
Patch benchmark · Independent grader
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.