How to use Claude to evaluate coding configuration
Config-drift-checker generated Claude Code evaluation cases and ran comparisons with and without project setup. The author found hook and skill effects, misleading pass reasons, grader errors and a real Docker teardown from an eval workspace.
news.ycombinator.com
Suggested prompt
Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.
Evaluate [Claude Code project setup] by generating cases from [CLAUDE.md], [skill] and [guard hook]. Run the real agent with and without the setup in [isolated test environment], including negative cases. Preserve grader reasons and responses, distinguish model refusals from hook enforcement, and rescore grader errors.
Reach people exploring AI agents
See placements →Evidence at a glance
Sourced account- Source author
- smartwordworld
- Original account
- Read the source ↗
- Published
- Source link checked
- Reported result
- The author found measurable guard-hook and conventions-skill effects, but also misleading pass reasons, grader errors and a Docker teardown that prompted a safety net.
- Human involvement
- The author inspected verdict reasons, identified grader errors and added a safety net after observing a destructive Docker side effect.
A linked account supports attribution. It does not mean we independently reproduced its result.
Report an issue with this case
Open correction emailOpens your email app with the case link included. Nothing is sent automatically.
How the agent did it
- Generated evaluation cases from the existing CLAUDE.md, conventions skill and guard hook.
- Ran the real agent in throwaway repository copies with and without the setup.
- Inspected grader verdicts and reasons, discovering model refusals before hook execution.
The author built config-drift-checker to determine whether a Claude Code setup continued behaving as intended after releases or configuration edits. The tool writes evaluation cases from an existing CLAUDE.md, conventions skill and guard hook, using Anthropic's plugin evaluation format. It runs the real agent in a throwaway repository copy, and a GitHub Action reruns cases on Claude Code releases and pull requests touching the setup.
The author compared runs with and without the setup on a small Spring Boot service. They report that the guard hook reliably stopped git reset --hard and that the conventions skill changed a three-file feature's output. A negative case confirmed the skill did not activate for a Dockerfile request.
Inspection of grader reasons revealed a limitation: a passing hook test could reflect the model's own refusal before the hook ran. Another run without the hook tore down a real Docker stack because a compose file pinned its project name, leading to a safety net in every run. The author also found grader errors involving quoted rules, negations and Java signatures and favored rescoring over rerunning. Who it's for: developers maintaining agent configuration who want repeatable behavioral evaluations and explanations behind the scores.
Prompt
Suggested prompt (written by Agents In Use, not the author's)
Source: Hacker News · smartwordworld
Frequently asked questions
Why compare runs with and without setup?
The author found this comparison useful for isolating the effects of a guard hook and conventions skill. A negative Dockerfile case checked that the skill did not fire outside its intended scope.
Why inspect reasons behind passing scores?
After one hook test passed, eight of the next nine runs showed the model refusing the command before the hook ran. The same score therefore had a different cause.
More details
Remaining steps
- Added a safety net after an evaluation run tore down a real Docker stack.
About this account
The real Claude Code agent executed evaluation cases; generated cases and CI reruns supported repeated configuration checks.
Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=49468946. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.
Credit: smartwordworld on Hacker News
Source published
Added to Agents In Use
Source checked
Tools and topics
Claude Code · config-drift-checker · GitHub Actions · Spring Boot · Docker
agent evaluations · configuration · CI
Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.