Claude Code tests literature-grounded training ideas
Paper Lantern reports two Claude Code autoresearch runs where literature access supplied training ideas; the resulting configuration had lower validation loss in the team’s own small-model test.
paperlantern.ai
Original prompt
Shared by the source author. See source notes for context.
What research exists on scaling batch size for short training runs?
Reach people exploring AI agents
See placements →What happened
Paper Lantern reports two Claude Code autoresearch runs where literature access supplied training ideas; the resulting configuration had lower validation loss in the team’s own small-model test.
Reported by Paper Lantern teamHow the agent did it
- Start equivalent autoresearch runs from the same model and conditions.
- Give one Claude Code run access to the Paper Lantern MCP tools.
- Search approaches and inspect implementation details before each idea.
Where a person stepped in
The team sets trial budgets and compares validation results. This is its own vendor-run small-model experiment, without independent replication or proof of larger-model gains.
Paper Lantern connects Claude Code to its research-paper MCP server for an autoresearch experiment. Two runs start with the same small GPT model and conditions. Each explores one hundred experiments with a five-minute training budget per trial and at most three hyperparameter-tuning attempts per idea.
In the literature-assisted run, the agent searches for candidate approaches, examines implementation details and tries ideas against the model. The team reports considering 520 papers and directly testing ideas from twenty-five. One useful example is batch-size reduction: paper-backed guidance supplies the learning-rate adjustment needed when halving the batch, rather than treating the batch setting in isolation.
Not every researched technique fits. The source identifies Dynamic Tanh and SeeDNorm as incompatible with the architecture. When the best configurations are trained for two hours, the team reports validation-loss values of 0.4624 without the paper server and 0.4475 with it. This is a vendor-run comparison on a roughly seven-million-parameter model, not independent evidence of a universal improvement or larger-model scaling. The practical workflow is to turn literature into testable implementation choices, keep failed trials visible and use measured validation results to decide which configuration survives.
Prompt
Original prompt
Source: Blog · Paper Lantern team
Frequently asked questions
How was each trial bounded?
The reported setup allowed five minutes of training per experiment and at most three tuning attempts per idea.
Did every paper-based idea work?
No. Dynamic Tanh and SeeDNorm were reported as incompatible with the tested architecture.
What does the loss comparison prove?
It is the team’s reported result for this small-model configuration and hardware, not an independently reproduced universal performance claim.
More details
Remaining steps
- Test each change within the stated trial budget and retain measured results.
- Compare longer training of the best configurations, including failures.
About this account
Claude Code searches literature through MCP, implements candidate training techniques, runs experiments and uses observed results to keep or discard changes.
Firsthand account by Paper Lantern team; original publication date verified against Primary blog March 26, 2026. Outcomes are the author’s reports, not independently reproduced. Original research query excerpt shown in the experiment; not the full autoresearch instruction.
Credit: Paper Lantern team
Source published 26 Mar 2026
Source checked 14 Oct 2026
Tools and topics
Claude Code · Paper Lantern MCP · autoresearch-macos · TinyStories · M4 Pro
Literature-grounded trials · Training experiments · Validation loss
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.