How to validate browser agent tasks
Skyvern added planning and goal validation to browser automation, then reported 85.85% accuracy in a cloud-based WebVoyager evaluation with documented dataset changes.
Original prompt
Shared by the source author. See source notes for context.
Navigate to Amazon.com and add an iPhone 16, a phone case, and a screen protector to cart
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- suchintan
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The authors report 85.85% WebVoyager accuracy after adding planning and validation, using cloud browsers and a dataset with obsolete tasks removed and booking dates updated.
- Human involvement
- The team changed and published the evaluation setup; the post does not describe human review of each browser action.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Added planning to decompose browser objectives into smaller goals.
- Added validation to check achieved goals and feed errors to the planner.
- Removed obsolete evaluation tasks and updated booking dates.
Skyvern's engineering post describes improving its browser agent from a single decision-and-action loop to a planner, actor and validator architecture. The example task was adding an iPhone 16, a case and a screen protector to an Amazon cart. With the original approach, a multi-item request could produce repeated phones or protectors instead of the intended combination.
The team added a planning phase to break complex objectives into smaller goals and retain working memory of completed and outstanding items. They report that WebVoyager performance rose from about 45% to about 68.7%. However, an executor could still report success after clicking an add-to-cart button even when the product had not been added.
A validation phase checked whether planner goals were actually achieved and sent errors or adjustments back to the planner. The post attributes a final 85.85% score to this change. Tests ran in Skyvern Cloud with asynchronous browsers and GPT-4o plus GPT-4o-mini. The team removed eight obsolete tasks, updated old booking dates, and published the evaluation run and dataset modifications.
These figures describe the team's modified benchmark, rather than every possible website workflow. Who it's for: browser automation teams that need explicit planning and outcome checks for multi-step tasks.
Prompt
Original prompt
Source: Blog · suchintan
Frequently asked questions
Why add a validator after planning?
Some executor actions were reported as successful even though the page outcome had failed, such as clicking add to cart without adding the product. Validation checks goals and reports corrections to the planner.
How was the benchmark changed?
The team removed eight tasks whose results were no longer valid and updated old flight and hotel booking dates in both prompts and answers.
More details
Remaining steps
- Ran evaluations in asynchronous Skyvern Cloud browsers.
- Published evaluation runs and dataset modifications.
About this account
Skyvern plans browser goals, executes actions and validates outcomes, returning failures to the planner for adjustments.
Platform: Blog. Attribution: https://blog.skyvern.com/skyvern-2-0-state-of-the-art-web-navigation-with-85-8-on-webvoyager-eval/. Prompt: Original (quoted from source). Sourced by the Agents In Use pipeline.
Credit: suchintan on Blog
Source published
Added to Agents In Use
Source checked
Tools and topics
Skyvern · Skyvern Cloud · GPT-4o · GPT-4o-mini · WebVoyager
Browser agents · Validation · Evaluation
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.