Claude crawls 60 documentation pages into clean Markdown
Jessica Doering had Claude use her Pulpie MCP server to crawl 60 documentation pages into Markdown in 49 seconds, including a cold backend start.
dev.to
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Use the documentation crawler to save this docs section as Markdown. Preserve code, tables, headings and source URLs, then confirm how many pages were retrieved and list any failures.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source poster / publisher
- Jessica Doering (@sizzlebop)
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The author reports 60 documentation pages saved as Markdown in about 49 seconds including cold startup.
- Human involvement
- Doering selected the test site, built the tool and reported the timings. The source does not describe a second-site validation or independent benchmark.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Doering built a local MCP server that extracts full web pages and documentation into Markdown.
- She asked Claude to crawl a 60-page documentation site while the backend and extraction model were cold.
- The crawler saved all 60 pages; she reported about 10 seconds for startup and 37 seconds for crawling and saving.
Jessica Doering built Pulpie MCP to let coding agents fetch web pages and documentation as full Markdown rather than summaries or clipped search results. The server runs a local content-extraction model and can read a page inline, save it as a Markdown file, crawl a documentation section into a folder, or check what has already been saved.
To test the workflow, Doering asked Claude to crawl a documentation site containing 60 pages. The backend was cold, so the test included starting the server and loading the extraction model onto her GPU. She reports about 10 seconds for the startup and 37 seconds to crawl and save the pages, or about 49 seconds total. The resulting files preserve page structure such as tables, links, images, code and headings. The source is the tool’s creator reporting a single run, not an independent comparison across sites or agents.
The copyable pattern is to expose a documentation crawler as a tool, let the agent retrieve whole reference pages, and persist them for later tasks. Keep the source URL with each file so the agent can cite where a detail came from. The author does not share the exact instruction given to Claude, so the prompt below is suggested by Agents In Use.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: DEV Community · Jessica Doering (@sizzlebop)
Frequently asked questions
What does Pulpie MCP do?
It gives an agent tools to read a page, save it as Markdown, crawl a documentation section and check previously saved pages.
How many pages did Claude crawl in the test?
Doering reports that Claude crawled and saved 60 pages in about 37 seconds, with a cold start bringing the total to about 49 seconds.
Was this benchmark independently repeated?
No. It is the tool creator’s report of one documentation crawl, including backend startup and local model loading.
More details
About this account
Claude invoked the MCP crawler to collect and save the documentation pages; the author built and started the local extraction backend.
Reviewed direct DEV article (HTTP 200) for author, date, tool description and a single 60-page run. Timings are the tool creator’s report, not an independently reproduced benchmark. No exact prompt shared.
Credit: Jessica Doering (@sizzlebop) — DEV Community
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude · Pulpie MCP · Pulpie extraction model
documentation · MCP · Markdown crawler
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.