How to use Claude to optimize local llama.cpp inference
An overnight Claude Code task produced reported llama.cpp performance patches with per-patch tests. The author used them locally for three weeks and shared a temporary fork.
news.ycombinator.com
Original prompt
Shared by the source author. See source notes for context.
Llama.cpp, fix the bugs that cause horrible speed decay in Qwen3.8-Flash-Next in longer context and with multiple users.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source poster / publisher
- Roark66
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The author reported 19 patches, up to tenfold speed gains in some local multi-user cases and three weeks without a crash, then shared a temporary fork.
- Human involvement
- The author tested through extended local use but did not fully understand the patches and withheld an upstream pull request.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Delegate long-context and multi-user inference slowdown investigation overnight.
- Authorize the agent to build llama.cpp and iterate on the local hardware.
- Receive patches with README explanations and before-and-after speed tests.
The author delegated a local inference performance problem overnight through Claude Code, describing the working model as Fable 5.1. Their request targeted severe speed decay in llama.cpp when running Qwen3.8-Flash-Next with longer contexts and multiple users. They authorized building the software, running it on their hardware and iterating.
By morning, the author reported 19 patches containing about 6,000 lines of code. Each patch had a README explanation and before-and-after speed tests. The claimed improvement reached up to tenfold in some cases involving five simultaneous users with long contexts. This was a reported local result, with no claim of compatibility across other models or hardware.
The author said the modified code had run for three weeks without a crash or failure. They subsequently used it in a setup where Fable distributed work to five Qwen agents. Rather than submit patches they had not fully understood, they published a temporary fork and posted in llama.cpp discussions.
Who it's for: experienced users investigating inference bottlenecks on their own hardware while keeping broader compatibility questions explicit.
Prompt
Original prompt
Source: Hacker News · Roark66
Frequently asked questions
Was the reported speedup universal?
No. The author reported up to tenfold improvement in some local cases with five simultaneous users and long contexts, and explicitly questioned other hardware and model compatibility.
Were the patches submitted upstream?
The author published a temporary fork and posted about it in llama.cpp discussions, rather than submitting the 19 patches as a pull request.
More details
Remaining steps
- Run the modified code locally for three weeks.
- Publish a temporary fork and discuss it with the llama.cpp community.
About this account
The agent was authorized to build, run and iterate overnight on the author's hardware without step-by-step direction.
Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=50056221. Prompt: Original (quoted from source). Sourced by the Agents In Use pipeline.
Credit: Roark66 on Hacker News
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude Code · Fable 5.1 · llama.cpp · Qwen3.8-Flash-Next
llama.cpp · Local inference · Performance · Qwen
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.