Agents In Use
What’s new todayLast update
47 use cases added today

How to evaluate code reviewers

A developer published an evaluation of Greptile, OpenQodex and shadowclone, reporting known-bug detection and model-judged finding quality while acknowledging bugs their own reviewer missed.

Source published news.ycombinator.comshadowclone
OUR ADAPTATION

Suggested prompt

Suggested prompt written by Agents In Use, not the source author. Replace bracketed inputs and review tool permissions first.

Evaluate [code review tools] on [pull requests with known bugs]. Record which known bugs each finds and have [judges] assess finding validity. Report detection and accepted-finding rates separately, including bugs [your reviewer] misses.

Evidence at a glance

Sourced account
Source poster / publisher
theonly1me
Original account
Read the source ↗
Published
Source link checked
Reported result
Greptile had the highest reported known-bug detection rate. Shadowclone had higher proportions of findings judged real but missed four real bugs Greptile caught.
Human involvement
The author designed and ran the comparison, inspected differences and publicly acknowledged bugs their own tool missed.

A linked account supports attribution. It does not mean we independently reproduced its result.

Report an issue with this caseOpen correction email

Opens your email app with the case link included. Nothing is sent automatically.

How the agent did it

  1. Added review capabilities to shadowclone for the comparison.
  2. Ran Greptile, OpenQodex and shadowclone against the evaluation.
  3. Ran local reviewers three times per pull request.

The author evaluated code review tools after repeatedly seeing an open-source product advertised as a replacement for CodeRabbit. That investigation also prompted them to add review capabilities to shadowclone, their tool for managing agent skills and engineering preferences.

They compared Greptile, OpenQodex and shadowclone's own reviewer. The shadowclone implementation included a code review skill, a local reviewer and a GitHub bot for cloud reviews. Local reviewers ran three times per pull request. The evaluation measured the proportion of known bugs found and the proportion of findings that model judges considered real.

The reported results exposed a tradeoff. Greptile found 69% of known bugs, with 68% of findings judged real. Shadowclone's cloud bot found 38%, with 96% judged real; its local reviewer found 36%, with 91% judged real. OpenQodex found 31%, with 86% judged real. Despite its higher proportion of accepted findings, shadowclone missed four real bugs Greptile caught.

The author published the results and an open-source harness that others could rerun, emphasizing the value of disclosing shortcomings.

Who it's for: Developers comparing automated reviewers on both bug detection and the credibility of their findings.

Prompt
Suggested prompt (written by Agents In Use, not the author's)

Source: Hacker News · theonly1me

Frequently asked questions

Which reviewer found the most known bugs?

Greptile found 69% of the known bugs in the author's evaluation, compared with 38% for shadowclone cloud, 36% for shadowclone local and 31% for OpenQodex.

Did shadowclone's higher accepted-finding rate mean it caught everything?

No. The author said shadowclone missed four real bugs that Greptile caught, despite a higher proportion of its findings being judged real.

Similar use cases

Works well with

More details

Remaining steps

  1. Compared known-bug detection with model judgments of findings.
  2. Published evaluation results and the rerunnable harness.

About this account

Automated reviewers produced findings, and model judges assessed their validity within the author's evaluation harness.

Platform: Hacker News. Attribution: https://news.ycombinator.com/item?id=50043719. Prompt: Suggested (written by Agents In Use, not the author's). Sourced by the Agents In Use pipeline.

Credit: theonly1me on Hacker News

Source published

Added to Agents In Use

Source checked

Tools and topics

shadowclone · Greptile · OpenQodex · GitHub · Model judges

Code review · Evaluation · Bug detection

Editorial completeness: 100/100. Measures documented fields, not agent performance or verified outcomes.