How to use Claude to evaluate coding configuration
Config-drift-checker generated Claude Code evaluation cases and ran comparisons with and without project setup. The author found hook and skill effects, misleading pass reasons, grader errors and a real Docker teardown from an eval workspace.
news.ycombinator.com
Prompt to try
Editorial adaptation, not the source author’s exact prompt. Replace bracketed inputs and review tool permissions first.
Evaluate [Claude Code project setup] by generating cases from [CLAUDE.md], [skill] and [guard hook]. Run the real agent with and without the setup in [isolated test environment], including negative cases. Preserve grader reasons and responses, distinguish model refusals from hook enforcement, and rescore grader errors.
Reach people exploring AI agents
See placements →What happened
The author found measurable guard-hook and conventions-skill effects, but also misleading pass reasons, grader errors and a Docker teardown that prompted a safety net.
Reported by smartwordworld on Hacker NewsHow the agent did it
- Generated evaluation cases from the existing CLAUDE.md, conventions skill and guard hook.
- Ran the real agent in throwaway repository copies with and without the setup.
- Inspected grader verdicts and reasons, discovering model refusals before hook execution.
Where a person stepped in
The author inspected verdict reasons, identified grader errors and added a safety net after observing a destructive Docker side effect.
A person approved each step before the agent continued.
The author built config-drift-checker to determine whether a Claude Code setup continued behaving as intended after releases or configuration edits. The tool writes evaluation cases from an existing CLAUDE.md, conventions skill and guard hook, using Anthropic's plugin evaluation format. It runs the real agent in a throwaway repository copy, and a GitHub Action reruns cases on Claude Code releases and pull requests touching the setup.
The author compared runs with and without the setup on a small Spring Boot service. They report that the guard hook reliably stopped git reset --hard and that the conventions skill changed a three-file feature's output. A negative case confirmed the skill did not activate for a Dockerfile request.
Inspection of grader reasons revealed a limitation: a passing hook test could reflect the model's own refusal before the hook ran. Another run without the hook tore down a real Docker stack because a compose file pinned its project name, leading to a safety net in every run. The author also found grader errors involving quoted rules, negations and Java signatures and favored rescoring over rerunning. Who it's for: developers maintaining agent configuration who want repeatable behavioral evaluations and explanations behind the scores.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Hacker News · smartwordworld
Frequently asked questions
Why compare runs with and without setup?
The author found this comparison useful for isolating the effects of a guard hook and conventions skill. A negative Dockerfile case checked that the skill did not fire outside its intended scope.
Why inspect reasons behind passing scores?
After one hook test passed, eight of the next nine runs showed the model refusing the command before the hook ran. The same score therefore had a different cause.
More details
Remaining steps
- Added a safety net after an evaluation run tore down a real Docker stack.
About this account
The real Claude Code agent executed evaluation cases; generated cases and CI reruns supported repeated configuration checks.
Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=49468946. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: smartwordworld on Hacker News
Source published 27 Aug 2026
Source checked 14 Oct 2026
Tools and topics
Claude Code · config-drift-checker · GitHub Actions · Spring Boot · Docker
agent evaluations · configuration · CI
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.