How to use Claude to evaluate a restricted planning skill
Claude evaluation subagents checked a planning skill after web-tool restrictions, preserving its file workflow while exposing the scope and cost of the test.
dev.to
Original prompt
Shared by the source author. See source notes for context.
I need to build a Python CLI tool that lets me add, list, and delete todo items.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- Othman Adi
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The restricted planning skill retained its three-file planning workflow in the published evaluation; its extra overhead and limited assertion scope are documented.
- Human involvement
- Adi investigated the scanner warning, chose the scope changes, and reviewed the evaluation results. The skill’s boundary asks users to confirm instruction-like external content.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Adi restricted declared web tools and separated external findings from the repeatedly reread goal file.
- Claude evaluation subagents executed paired tasks with and without the restricted skill.
- Graders and blind comparators checked planning artifacts, while the author reported the added overhead.
Othman Adi’s planning-with-files skill stored goals, findings, and progress in separate Markdown files. A hook repeatedly reread the goal file before tool calls. That useful reminder could also amplify untrusted instructions if fetched web content entered the same file.
Adi removed declared web-fetch and web-search tools across the supported IDE variants and documented a boundary: external material belonged in findings, while instruction-like content needed user confirmation. He then used Claude Code’s skill-creator framework to check whether restricting the skill had damaged its core planning workflow.
Executor, grader, comparator, and analyzer subagents ran paired tasks with and without the skill. The published examples included a persistent todo command-line tool, FastAPI debugging, and a Django migration plan. Assertions checked the intended file structure and planning sections; blind comparisons assessed the outputs. The restricted version continued to produce the prescribed planning artifacts, with additional token and time overhead.
The scope of that evidence matters. File-contract compliance and model comparisons support workflow preservation. They do not establish immunity to prompt injection or prove a general improvement in software quality. The account joins a concrete security-scope change to a reproducible check of the feature it was meant to preserve.
Prompt
Original prompt
Source: DEV Community · Othman Adi
Frequently asked questions
What could amplify an injected instruction?
The hook reread the goal file before subsequent tool calls.
What did the evaluation actually measure?
Planning-file contracts and paired output comparisons, not exploit immunity.
Where should fetched content go?
The documented boundary places it in findings rather than the goal file.
More details
About this account
Adi restricted declared web tools and separated external findings from the repeatedly reread goal file. Claude evaluation subagents executed paired tasks with and without the restricted skill.
Original evaluation prompt excerpt, not the security-remediation request. Opened article and repository docs/evals.md. Assertions heavily reward the skill’s file structure; reported pass rates are not security assurance. Original prompt excerpt.
Credit: Othman Adi — DEV Community
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude Code · skill-creator · Markdown · A/B evaluations
skill security · prompt injection · evaluation
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.