How to compare coding models in a Pi agent loop
A developer used a custom Pi subagent extension to run coding models against five specifications. Each agent could read, write, edit and run shell commands, while separate hidden checks graded the saved solution files.
reddit.comSuggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
In a fresh directory, read [TASK.md] and implement solution.py plus test_solution.py. Use read, write, edit and shell tools to test your work. Stop within [time limit] and report actual files and test results. Keep the external grader unavailable to the agent. As evaluator, score the files on disk even after a timeout and record failed checks, token use and ambiguous requirements.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source poster / publisher
- u/Ok_Substance2327
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The source reports two models passing all five tasks, alongside failed files, timeouts and false completion claims. The author later notes that one specification was ambiguous and that a model was added to the post. Results are self-reported and the complete reproducible setup was not supplied.
- Human involvement
- The author wrote specifications and external checks, set run limits and interpreted results. They acknowledge an ambiguous pattern-matching rule and variable timing near a performance threshold. The full setup was not yet shared.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- The author created five coding specifications and gave each model-task run a fresh directory containing only TASK.md.
- A custom Pi subagent extension let each model read, write, edit and execute shell commands to produce a solution and tests.
- A separate checker graded the saved files using fixed cases, randomised cases and performance gates after completion or the time cap.
A LocalLLaMA poster compared coding models through a tool-using agent loop, not a collection of one-shot code answers. An author comment identifies the runner as a custom Pi subagent extension. Each model-task run began in a fresh directory containing only TASK.md and could read, write, edit and execute shell commands.
The five tasks covered CSV quoting, arithmetic evaluation, shell tokenisation, interval scheduling and pattern matching. Agents had to write a solution and a test file. A separate checker, unavailable to the agent, used fixed examples, randomised cases and performance gates. If a run exceeded its cap, the checker graded whatever files existed.
The source reports two models passing all five tasks. It also describes agents claiming files existed when they did not, spending a run rereading a specification, or continuing a costly fuzz test after reaching a correct solution. Those failure accounts are more informative than a success-only table.
The comparison has limits. The title names nine models, while the edited body lists ten; a returned author comment says a coder model was added. The author also acknowledges ambiguity in the pattern-matching specification, and one interval-scheduling result varies around its timing threshold. The complete reproducible setup was not yet supplied.
Who it's for: developers designing agent evaluations that separate an agent's own test claims from independent checks on actual saved files.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Reddit · u/Ok_Substance2327
Frequently asked questions
What agent runner did the author identify?
In returned comments, the author says the tests ran through their own Pi subagent extension. Each run had read, write, edit and shell tools in a fresh directory.
Why should the reported scores be treated cautiously?
The author acknowledges an ambiguous pattern-matching specification and timing variation near one performance gate. The edited post adds a model, and the full reproducible setup was not yet supplied.
More details
Remaining steps
- The author compared results and token use, then disclosed specification ambiguity and timing instability in the post and returned comments.
About this account
Each model ran a tool-using agent loop through the author's Pi subagent extension, with read, write, edit and shell access. The loop attempted implementation and self-testing without access to the external grader.
Firsthand account by u/Ok_Substance2327: https://www.reddit.com/r/LocalLLaMA/comments/1x8qlvw/nine_models_five_coding_tasks_one_16gb_gpu/. Full public RSS post and returned comments captured on 19 October 2026. Body eligibility passed at 0.88. Outcomes are author-reported, not independently reproduced. Suggested prompt by Agents In Use, not the author's original prompt.
Credit: u/Ok_Substance2327 on Reddit
Source published
Added to Agents In Use
Source checked
Tools and topics
Pi · Custom subagent extension · Python · Hidden checker
agent evaluation · coding models · hidden tests · failure analysis
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.