How to use Claude to configure reusable web scrapers
ScrapAI delegates site analysis and extraction-rule setup to an agent, then uses validated JSON configurations for recurring Scrapy collection without per-page AI.
news.ycombinator.com
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Using [ScrapAI project], add [website] to [collection project]. Analyze the site, write URL patterns and extraction rules as JSON, test them on five pages, and validate the configuration before saving it for subsequent Scrapy runs.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- iranu
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The author reported production use across more than 500 websites, with reusable JSON configurations and no per-page inference calls.
- Human involvement
- The post describes Pydantic validation but does not describe human review of each configuration.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Ask the agent to add a website to a scraping project.
- Analyze the website and write URL patterns and extraction rules.
- Test the extraction configuration on five pages.
The ScrapAI author described a research group's need to collect online communications from news sites, blogs, forums, and policy organizations. Maintaining individual scrapers became burdensome as the number of sites grew: redesigns broke existing scrapers, and new sites required more implementation work. The author reported using the resulting system in production across more than 500 websites.
In the reported workflow, a user asks an agent to add a site to a project. The agent analyzes that site, writes URL patterns and extraction rules, tests them on five pages, and saves a JSON configuration to a database. After configuration, Scrapy performs collection without AI in the per-page loop. The author recommended Claude Code with Sonnet 4.5, while saying other agents capable of reading instructions and running shell commands also worked.
The agent writes JSON configurations rather than Python code. Pydantic validates those configurations before they reach the database. The implementation also uses SQLAlchemy and Alembic. The post describes browser-based challenge handling separately from the agent's configuration task.
Who it's for: Research teams maintaining recurring collections across many websites who want reusable extraction configurations.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Hacker News · iranu
Frequently asked questions
Does the agent run on every scraped page?
No. It analyzes a site, writes rules, tests five pages, and saves a JSON configuration. Scrapy then handles collection without per-page AI inference.
How are configurations checked?
The author says the agent writes JSON rather than Python, and the JSON passes through Pydantic validation before touching the database.
More details
Remaining steps
- Validate the JSON configuration and save it to the database.
- Run subsequent collection through Scrapy without per-page AI.
About this account
The agent handles site configuration and testing; subsequent page collection runs in Scrapy without an AI loop.
Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=47233229. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: iranu on Hacker News
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude Code · Sonnet 4.5 · ScrapAI · Scrapy · Pydantic · SQLAlchemy · Alembic
Software development
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.