How to use Claude to configure reusable web scrapers
ScrapAI delegates site analysis and extraction-rule setup to an agent, then uses validated JSON configurations for recurring Scrapy collection without per-page AI.
news.ycombinator.com
Prompt to try
Editorial adaptation, not the source author’s exact prompt. Replace bracketed inputs and review tool permissions first.
Using [ScrapAI project], add [website] to [collection project]. Analyze the site, write URL patterns and extraction rules as JSON, test them on five pages, and validate the configuration before saving it for subsequent Scrapy runs.
Reach people exploring AI agents
See placements →What happened
The author reported production use across more than 500 websites, with reusable JSON configurations and no per-page inference calls.
Reported by iranu on Hacker NewsHow the agent did it
- Ask the agent to add a website to a scraping project.
- Analyze the website and write URL patterns and extraction rules.
- Test the extraction configuration on five pages.
Where a person stepped in
The post describes Pydantic validation but does not describe human review of each configuration.
A person set the goal and reviewed the result.
The ScrapAI author described a research group's need to collect online communications from news sites, blogs, forums, and policy organizations. Maintaining individual scrapers became burdensome as the number of sites grew: redesigns broke existing scrapers, and new sites required more implementation work. The author reported using the resulting system in production across more than 500 websites.
In the reported workflow, a user asks an agent to add a site to a project. The agent analyzes that site, writes URL patterns and extraction rules, tests them on five pages, and saves a JSON configuration to a database. After configuration, Scrapy performs collection without AI in the per-page loop. The author recommended Claude Code with Sonnet 4.5, while saying other agents capable of reading instructions and running shell commands also worked.
The agent writes JSON configurations rather than Python code. Pydantic validates those configurations before they reach the database. The implementation also uses SQLAlchemy and Alembic. The post describes browser-based challenge handling separately from the agent's configuration task.
Who it's for: Research teams maintaining recurring collections across many websites who want reusable extraction configurations.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Hacker News · iranu
Frequently asked questions
Does the agent run on every scraped page?
No. It analyzes a site, writes rules, tests five pages, and saves a JSON configuration. Scrapy then handles collection without per-page AI inference.
How are configurations checked?
The author says the agent writes JSON rather than Python, and the JSON passes through Pydantic validation before touching the database.
More details
Remaining steps
- Validate the JSON configuration and save it to the database.
- Run subsequent collection through Scrapy without per-page AI.
About this account
The agent handles site configuration and testing; subsequent page collection runs in Scrapy without an AI loop.
Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=47233229. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: iranu on Hacker News
Source published 3 Mar 2026
Source checked 14 Oct 2026
Tools and topics
Claude Code · Sonnet 4.5 · ScrapAI · Scrapy · Pydantic · SQLAlchemy · Alembic
Software development
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.