How to use Devin to benchmark MCP overhead
Devin built a repeatable MCP latency benchmark with direct-call baselines, percentile reports and a scheduled workflow using synthetic secrets for local runs.
github.com
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Add an end-to-end MCP latency benchmark before changing runtime behavior. Measure cold and warm health, listing, execute, and search calls against direct HTTP/MCP baselines; report percentiles and overhead in JSON and job summaries. Use synthetic secrets, bounded runs, and configuration skips, distinguishing local Miniflare measurements from production claims.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- devin-ai-integration[bot]
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- An end-to-end MCP benchmark command, statistics helpers and weekly/manual reporting workflow merged; example timings are local measurements.
- Human involvement
- The public PR records task constraints and automated review or checks. Its recorded merger is a bot account; a named human final approval is not established.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- The agent added end-to-end timings with cold/warm samples and direct-call baselines.
- It implemented percentile summaries, overhead subtraction and JSON reports.
- It wired weekly/manual runs with configuration skips and synthetic-secret rules.
Devin added a repeatable latency benchmark to Kody before changing its runtime. Existing handler timings excluded authentication, database lookups, leases, Durable Object calls and serialization, so they could not show the full overhead a client experienced.
The new command connected a stateless MCP client and measured cold and warm calls across health, tool listing, execute variants and search. Direct HTTP fetch and direct MCP calls supplied comparison baselines. The output included percentile tables and a separate calculation of overhead above those direct paths, with JSON export for later analysis.
A weekly GitHub workflow and manual dispatch exposed the measurements in job summaries and uploaded reports. Missing origin or token configuration caused a skip rather than a misleading failed benchmark. Documentation required synthetic secrets and refused password-based login against the production site, directing that environment to a dedicated token.
The agent reported local benchmark runs, statistics tests, type checking and repository hooks. The pull request merged through kody-bot on October 11. Its example numbers came from Miniflare and local targets, not production. The useful artifact was the measurement command and workflow; the source explicitly scoped this phase to measurement without runtime changes.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: GitHub · devin-ai-integration[bot]
Frequently asked questions
Why were handler timings insufficient?
They excluded authentication, database and orchestration work outside the handler.
Were example results production measurements?
No. The PR labels its sample measurements local Miniflare results.
What happens without credentials?
The scheduled workflow skips when its origin or token is unconfigured.
More details
About this account
The new command connected a stateless MCP client and measured cold and warm calls across health, tool listing, execute variants and search. Direct HTTP fetch and direct MCP calls supplied comparison baselines. The output included percentile tables and a separate calculation of overhead above those direct paths, with JSON export for later analysis.
Primary agent-authored PR and public API opened. Results and checks are reported by the PR, not independently reproduced. Bot merge is not evidence of human review. Prompt is suggested by Agents In Use, not a quoted original.
Credit: devin-ai-integration[bot] on GitHub
Source published
Added to Agents In Use
Source checked
Tools and topics
Devin · GitHub · Cloudflare Workers
Repository maintenance · Recorded checks
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.