How to evaluate website builds with Unknown
Ploy corrected evaluation assumptions, reran homepage redesign comparisons, and examined tool-call failures before adopting GPT-5.6 Sol for its website agent.
ploy.aiPrompt to try
Editorial adaptation, not the source author’s exact prompt. Replace bracketed inputs and review tool permissions first.
Evaluate [challenger model] against [incumbent model] on [website fixture workspaces]. Inspect failure traces for harness assumptions, check parallel calls and batched reads, then rerun reference-design comparisons. Report cost, time, visual checks, and unresolved tool-call issues.
Reach people exploring AI agents
See placements →What happened
Ploy reports making GPT-5.6 Sol the default for every workspace after completed builds ran 2.2 times faster and cost 27% less in its corrected evaluation.
Reported by brryant on BlogHow the agent did it
- Triaged evaluation failures against full agent traces.
- Corrected harness assumptions and reran the redesign suite.
- Investigated code-tool arguments using production file-read traces.
Where a person stepped in
The team triaged failures against full traces and used visual, content, tool-trajectory, and file checks.
A person approved each step before the agent continued.
Ploy reported migrating its marketing website agent from Claude Opus 4.8 to GPT-5.6 Sol after a head-to-head evaluation. The agent plans pages, reads code, writes components, generates imagery, and takes screenshots of its work. The migration exposed assumptions in the surrounding software as well as differences between models.
The team first examined failures in its evaluation harness. Tool-call budgets favored sequential calls, while the challenger made parallel calls. The executor also lacked support for batched file reads. Another problem involved an implicit minimum score that rejected otherwise defensible designs. After correcting the harness, Ploy reran a homepage redesign suite using reference designs and visual checks.
The reported completed builds averaged 8 minutes and $3.06 for Opus, compared with 3 minutes 42 seconds and $2.22 for GPT-5.6. Ploy also investigated unused tool arguments that caused empty file reads, and reported improving design steering until the challenger met its brand-adherence requirement. The supplied excerpt ends during the tool-call discussion, so it does not provide the complete implementation fix.
Who it's for: Teams evaluating replacement models for an existing website-building agent.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Blog · brryant
Frequently asked questions
What did Ploy fix before comparing models?
It corrected harness assumptions involving tool-call budgets, batched file reads, and an implicit minimum-score threshold before rerunning the redesign suite.
What results did Ploy report?
Mean completed builds took 8 minutes and cost $3.06 with Opus 4.8, versus 3 minutes 42 seconds and $2.22 with GPT-5.6. These are the post’s results for its own workload.
More details
About this account
The production agent plans and builds pages, takes screenshots, and decides when its work is complete.
Platform: Blog. Attribution: https://ploy.ai/blog/migrating-a-production-ai-agent-to-gpt-5-6. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: brryant on Blog
Source published 12 Jul 2026
Source checked 14 Oct 2026
Tools and topics
Ploy agent · Vercel AI SDK
Model evaluation · Website building
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.