How to evaluate website builds
Ploy corrected evaluation assumptions, reran homepage redesign comparisons, and examined tool-call failures before adopting GPT-5.6 Sol for its website agent.
ploy.aiSuggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Evaluate [challenger model] against [incumbent model] on [website fixture workspaces]. Inspect failure traces for harness assumptions, check parallel calls and batched reads, then rerun reference-design comparisons. Report cost, time, visual checks, and unresolved tool-call issues.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- brryant
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- Ploy reports making GPT-5.6 Sol the default for every workspace after completed builds ran 2.2 times faster and cost 27% less in its corrected evaluation.
- Human involvement
- The team triaged failures against full traces and used visual, content, tool-trajectory, and file checks.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Triaged evaluation failures against full agent traces.
- Corrected harness assumptions and reran the redesign suite.
- Investigated code-tool arguments using production file-read traces.
Ploy reported migrating its marketing website agent from Claude Opus 4.8 to GPT-5.6 Sol after a head-to-head evaluation. The agent plans pages, reads code, writes components, generates imagery, and takes screenshots of its work. The migration exposed assumptions in the surrounding software as well as differences between models.
The team first examined failures in its evaluation harness. Tool-call budgets favored sequential calls, while the challenger made parallel calls. The executor also lacked support for batched file reads. Another problem involved an implicit minimum score that rejected otherwise defensible designs. After correcting the harness, Ploy reran a homepage redesign suite using reference designs and visual checks.
The reported completed builds averaged 8 minutes and $3.06 for Opus, compared with 3 minutes 42 seconds and $2.22 for GPT-5.6. Ploy also investigated unused tool arguments that caused empty file reads, and reported improving design steering until the challenger met its brand-adherence requirement. The supplied excerpt ends during the tool-call discussion, so it does not provide the complete implementation fix.
Who it's for: Teams evaluating replacement models for an existing website-building agent.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Blog · brryant
Frequently asked questions
What did Ploy fix before comparing models?
It corrected harness assumptions involving tool-call budgets, batched file reads, and an implicit minimum-score threshold before rerunning the redesign suite.
What results did Ploy report?
Mean completed builds took 8 minutes and cost $3.06 with Opus 4.8, versus 3 minutes 42 seconds and $2.22 with GPT-5.6. These are the post’s results for its own workload.
More details
About this account
The production agent plans and builds pages, takes screenshots, and decides when its work is complete.
Platform: Blog. Attribution: https://ploy.ai/blog/migrating-a-production-ai-agent-to-gpt-5-6. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: brryant on Blog
Source published
Added to Agents In Use
Source checked
Tools and topics
Ploy agent · Vercel AI SDK
Model evaluation · Website building
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.