How to test Python changes with Claude Code
Fair-Rain3366 benchmarked Claude Code with and without Agentic Verification on 14 upstream Python commits. The plugin caught more mutants but missed one actual change and increased median runtime.
reddit.com
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Generate tests for [Python change] using [verification plugin]. Run deliberate mutants in disposable copies, compare against the old code without stashing my changes, and report survivors, change detection and runtime separately. Do not equate caught mutants with specification correctness.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- u/Fair-Rain3366
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The author reports mutation catches of 99% with the plugin versus 92% without it, but actual-change detection of 13/14 versus 14/14. Median runtime was 1.6 versus 0.4 minutes. These are maintainer-reported benchmark results.
- Human involvement
- The maintainer selected the benchmark and analysed transcripts. The comparison includes a missed change and does not establish general code correctness.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Configure Claude Code with the Python verification plugin and a baseline without it.
- Withhold developer tests and ask both arms to test the same upstream changes.
- Run mutants in disposable project copies and have Claude strengthen tests.
Reddit author Fair-Rain3366 maintains Agentic Verification, a Claude Code plugin for Python and pytest. They benchmarked Sonnet with and without the plugin on 14 real commits from seven established Python libraries. The developers' new tests were withheld, and the experimental design was registered before data collection.
The plugin runs deliberately broken variants of changed lines in disposable project copies. It reports surviving mutants and uncovered lines, then asks Claude to add tests and repeat the checks. Its base-check also compares tests against the old code. A verified verdict means the assessed mutants were caught, not that the implementation meets its specification.
The author reports 99% of mutants caught with the plugin versus 92% without it. However, the main change-detection result went the other way: 13 of 14 changes detected with the plugin, compared with all 14 without it. Median runtime increased from 0.4 to 1.6 minutes. Many extra catches concerned loose error-message assertions.
The experiment exposed Claude stashing user changes during comparisons. The author added base-check in a throwaway copy after examining that behaviour. A separate audit also prompted a held-out check before recommending test removals. These are the maintainer's results, not an independent replication. Who it's for: Python developers evaluating whether mutation-guided agent tests justify their extra runtime.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Reddit · u/Fair-Rain3366
Frequently asked questions
What did the plugin improve?
The maintainer reports higher mutation catches, but lower actual-change detection and longer median runtime in this 14-commit comparison.
Are the reported results independently verified?
No. This entry describes the original author's account and preserves its limitations. Agents In Use has not reproduced the workflow.
More details
Remaining steps
- Compare change detection, mutation catches and runtime, including the unfavourable result.
About this account
Claude generated tests, used mutation feedback and reran checks. The plugin isolated mutations and old-code comparisons in disposable copies.
Firsthand builder account: https://www.reddit.com/r/ClaudeCode/comments/1x7bxxl/i_benchmarked_my_claude_code_verification_plugin/. Full public RSS post and returned comments saved on 16 October 2026. Body eligibility passed at 0.98. Outcomes are author-reported. Suggested prompt by Agents In Use, not the author's original prompt.
Credit: u/Fair-Rain3366 on Reddit
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude Code · Agentic Verification · Python · pytest
mutation testing · benchmarking · test generation
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.