Codex builds a rate limiter for an over-crawled site
When crawlers hammered datasette.io, Willison used Codex to build a configurable IP rate-limiting plugin. His release note includes the configuration used on the live site.
simonwillison.netWhat happened
Willison published the plugin and says the included configuration is running on datasette.io; no measured traffic reduction is given.
Reported by Simon WillisonHow the agent did it
- Identify the areas receiving excessive crawler requests.
- Delegate implementation of configurable per-IP request limits to Codex.
- Configure exempt paths and rule-specific windows for the deployed plugin.
Where a person stepped in
Willison selected the production configuration. The note does not document a separate approval or review protocol.
A person approved each step before the agent continued.
More details
About this account
Codex implemented the plugin from the operational goal; the short release account does not expose every intermediate decision.
Firsthand release note links code and production configuration. Intermediate testing and effectiveness are not documented here.
Credit: Simon Willison
Added 14 Oct 2026
Source published 14 May 2026
Source checked 14 Oct 2026
Tools and topics
Codex · GPT-5.5 · Datasette
Operations · Checkpoints
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.
Prompt to try
Editorial adaptation, not the source author’s exact prompt. Replace bracketed inputs and review tool permissions first.
Inspect sanitized request logs for [service]. Propose route-specific rate limits and exemptions, then implement them on a branch with checks for normal traffic, bursts and proxy IP handling. Produce configuration and rollback instructions. Stop before enabling limits in production.