code-review-graph Argues Against Its Own Headline Number, and That's the Reason to Trust It

A local code intelligence graph that cuts the tokens your AI coding tools burn on reviews. The graph is fine. The honest benchmark writing wrapped around it is the part worth stealing.

The README opens with a big number. It says code-review-graph delivers a median 82x reduction in the tokens your AI assistant burns, with a best case of 528x. Then, a few lines down, it does something almost no trending repo does. It tells you that number is measured against a baseline no real agent ever pays, and it built a second, harsher benchmark to prove it.

I have read a lot of trending GitHub READMEs this year. The genre runs on inflation. Star counts pumped by momentum, savings figures measured against a strawman, benchmarks that grade the tool against the worst possible alternative so the multiplier looks enormous. code-review-graph does the opposite in public, and after watching a dozen code-graph projects trend on the exact same "we cut your tokens" pitch, the self-correction is what made me stop and actually read it.

What it is, quickly, because you have seen the shape before

The tool solves a real annoyance. AI coding assistants re-read large chunks of your codebase every time you ask them to review a change, and every re-read is tokens you pay for. code-review-graph parses your repo with Tree-sitter into an abstract syntax tree, stores it as a graph of nodes (functions, classes, imports) and edges (calls, inheritance, test coverage), and serves precise slices to your assistant over MCP. When a file changes, the graph traces every caller, dependent, and test that could be touched. That set is the change's blast radius, and your agent reads only those files instead of scanning the project.

If that sounds familiar, it should. Codebase-to-graph projects have been trending for months. The mechanism is not the novelty, and code-review-graph does not pretend it is. What separates this one is how it talks about its own results.

The headline number, and the sentence that undercuts it

Here is the move, in the maintainer's own framing. The 82x median comes from comparing the graph's per-question token cost against a whole-corpus baseline, meaning the cost of feeding the entire codebase to the model. The README says plainly that this baseline is "an upper bound no real agent pays: a competent agent greps for identifiers and reads only the best-matching files."

Most projects would stop at 82x and let you assume the best. This one built an agent_baseline evaluation that measures against the realistic alternative: a plain grep over the corpus, top three files by match count, token-counted and compared against the graph's query cost. That is the comparison that matters, because that is what a decent coding agent actually does today without any graph at all.

The repo's own description carries the more honest framing: roughly 6.8x fewer tokens on reviews and up to 49x on daily coding tasks. Those numbers are smaller than 82x and far smaller than 528x, and they are the ones I would plan around, because they are measured against a baseline that resembles reality. The gap between the 82x hero number and the 6.8x realistic one is not a lie. It is the maintainer showing you both and telling you which is which. That is rarer than it should be.

It ships a "known weaknesses" section, and means it

Scroll to the benchmarks and there is a heading most repos would never write: "Limitations and known weaknesses." The entries are specific enough to be uncomfortable, which is how you know they are real.

The impact analysis reports a recall of 1.0, and the README immediately flags why you should not be impressed: "the historical ground truth comes from the same graph edges the predictor walks, so it is an upper bound by construction." In plain terms, the tool is graded against its own map, so of course it finds everything on that map. Circular. They say so, and they run an honest co-change mode alongside it that grades predictions against the files a developer actually touched in the same commit, and they warn those numbers will be "substantially lower."

There is more. On trivial single-file edits, the graph context can cost more tokens than just reading the file, because the structural metadata carries overhead. Keyword search has a mean reciprocal rank of 0.35, which the README admits means ranking "needs improvement." Flow detection sits at 33 percent recall and only works reliably on Python. Each of these is a reason someone might not adopt the tool, printed by the person who wants you to adopt the tool.

I keep coming back to that. The failure modes are documented by the author, in the sales document, next to the install command.

Putting this into practice

The install is genuinely low-friction, which helps, because the honest thing to do with a tool like this is test it rather than trust the graph in the abstract.

pip install code-review-graph
code-review-graph install    # auto-detects and configures your AI coding tools
code-review-graph build      # parse your codebase

The install step detects which assistants you run (Claude Code, Codex, Cursor, Windsurf, Gemini CLI, Copilot, and a long list of others), writes the right MCP config for each, and injects graph-aware instructions into your platform rules. A 500-file project builds in about 10 seconds. After that, incremental updates re-parse only changed files in under two seconds. You need Python 3.10 or newer, and it is MIT licensed, so there is no trap in trying it.

The useful judgment is knowing where it pays off. Point it at a large repo and ask for a review, and the token savings are real against the grep-and-read baseline the tool honestly benchmarks itself on. Point it at a trivial two-line edit and it may cost you more than a plain file read, which the README tells you outright. So the practical rule is simple: use it on big repos and multi-file review tasks, skip it for small isolated changes.

If you want to trust the numbers rather than take them, the maintainer left you the door. There is a docs/REPRODUCING.md and the eval harness is in the repo. You can add --verify to cross-check displayed token counts against OpenAI's tokenizer with tiktoken installed. Run it on your own codebase and your own questions, because the six repos in the benchmark are not your repo.

Where I would stay skeptical, honesty and all

An honest README is a reason to trust the tool more than its neighbors. It is not proof the tool works for you, and I would hold two things in mind.

First, honest self-reporting is still self-reporting. The maintainer ran the benchmarks, chose the six repos, and wrote the methodology. The transparency is real and I credit it, but the right response to "here are my numbers and here is why the flattering one is misleading" is still to reproduce it on your own code, not to swap one set of trusted numbers for another. The candor lowers my guard less than it earns my respect.

Second, the graph has the failure mode every persisted index has: it goes stale. A graph that has drifted from the current code feeds your agent a confident, outdated picture, and a wrong map read with authority is worse than no map. Watch mode and hooks exist to keep it fresh, but that is a thing you have to keep working, not a thing that maintains itself.

And the usual trending-board caveat applies. The star count sits around 18.7k on the repo page and higher on some trackers, which is momentum, not a verdict on whether it survives past this month. Code-graph projects trend and vanish on a regular cycle. Broad language support (Python, JavaScript, Go, Rust, Java, and a long tail down to Solidity and GDScript) is a point in its favor for longevity, but longevity is earned in maintenance, not in a launch week.

Here is what I would actually take from this repo, whether or not you install it. The next time a trending tool hands you a 500x multiplier, look for the sentence that tells you what baseline it is measured against. If that sentence is missing, supply it yourself and watch the number shrink. code-review-graph wrote that sentence for you. Most won't, and now you know what its absence is hiding.

Sources: code-review-graph on GitHub and its README; PyPI project page. Benchmark figures are the maintainer's own, reproducible via the repo's eval harness; verify on your codebase before relying on them.