Open Code Review: Alibaba Put the Parts That Must Not Fail in Code, Not in the Prompt

Alibaba open-sourced the AI code reviewer it ran internally for two years. The interesting part is everything it refuses to let the model decide.

A code review tool that admits, in its own README, that it finds fewer bugs than Claude Code is an unusual thing to hit number two on Trendshift.

That is where alibaba/open-code-review sat on the afternoon of September 16, at version 1.12.4 (released the same day) and roughly 31,000 stars by the cache-busted shields badge. The README's benchmark note says the tool's recall is lower than a general-purpose agent's, and calls that "a deliberate trade-off favoring precision over noise." It also says the tool reaches higher precision and F1 than Claude Code on the same underlying model while using about one ninth of the tokens. Read together, those two sentences describe a design decision rather than a weakness, and the decision is the reason this repo is worth your time.

Open Code Review is not a better prompt for reviewing code. It is an argument that the steps of a review which must not go wrong should never have been the model's job.

What the model is not allowed to do

The README names the three failures anyone who has used a general-purpose agent for review will recognize. Incomplete coverage: on a large changeset the agent selectively reviews some files and skips others. Position drift: the reported line numbers and file references wander off the actual code. Unstable quality: natural-language skills are hard to debug, and review quality swings with small prompt changes. Alibaba's diagnosis is one sentence: "a purely language-driven architecture lacks hard constraints on the review process."

So the tool splits the work. Four things are done by deterministic code, not by the model:

File selection decides exactly which changed files get reviewed and which are filtered, so nothing important is dropped by an agent that got bored.

File bundling groups related files into one review unit (the README's example is a pair of message_en.properties and message_zh.properties files) and runs each bundle as a sub-agent with its own isolated context. That is divide and conquer, it stays stable on very large diffs, and it makes concurrent review a natural consequence rather than a feature.

Rule matching is template-engine based. Rules are matched to each file's characteristics before the model sees it, which keeps the prompt focused and, in Alibaba's words, is "more stable and predictable" than asking the model to decide which rules apply. The project's own description lists rules for null-pointer exceptions, thread safety, XSS, and SQL injection across multiple languages.

Positioning and reflection are external modules: an independent comment-positioning step that fixes where a finding points, and a reflection step that checks what it says.

The model gets what is left: scenario-tuned prompts and a purpose-built toolset. The README says the toolset was distilled from tool-call traces in large-scale production data (call frequency, per-tool repetition, the effect of adding a tool on the whole call chain), which is a claim I cannot verify but which describes exactly the kind of work most agent builders skip. The agent can read whole files, search the codebase, and inspect other changed files for context. It writes the comments. It does not decide what to look at.

The reason this matters beyond code review is that it is the same conclusion IBM Research published on September 15 from the other direction: agents flip on near-tied decisions, and a bigger model does not fix flat decisions. Alibaba's answer is to stop asking the model to make them.

What it looks like to run

The tool is a Go CLI distributed through npm. Installation is one line, and you need Git 2.41 or newer:

npm install -g @alibaba-group/open-code-review
ocr config provider   # pick a built-in provider or add a custom endpoint
ocr config model      # pick a model for that provider

The interactive setup walks through provider, API key, and model, then tests connectivity. The repo advertises OpenAI- and Anthropic-compatible endpoints, so a local model behind a compatible server should work, though I have not tested that path.

Review modes map onto how you actually work:

ocr review                                   # everything staged, unstaged, and untracked
ocr review --from main --to feature-branch   # merge-base diff of a branch
ocr review --commit abc123                   # one commit
ocr scan --path internal/agent               # whole files, no diff needed
ocr review --format json --output result.json

ocr scan is the one to notice. It reviews entire files rather than a diff, which turns the tool into an audit pass for a codebase you inherited or a directory that has no meaningful git history. Long reviews are resumable through ocr session list and --resume <session-id>.

Then there is delegation mode, which I think is the most interesting design choice in the repo:

ocr delegate preview
ocr delegate rule src/main.go src/handler.go

In delegation mode Open Code Review does the deterministic half (file selection, rule resolution) and hands the result to whatever coding agent you are already running. Claude Code, Codex, or Cursor performs the review with its own model and its own subscription; no OCR API key is required. There are plugins for each, plus OpenCode and a "skill-compatible agents" path, and an MCP server so the review agent can reach external tools. Alibaba is, in effect, selling you only the part it thinks the model is bad at and letting you keep your model.

The rest of the surface is what you would expect from something that ran inside a large company: CI integration for GitHub Actions, GitLab CI, GitFlic CI, and Gerrit; a session viewer to replay reviews in a browser and mark comments fixed or ignored; OpenTelemetry for observability; and custom review rules with path filtering. The docs live at open-codereview.ai rather than in the repo.

Put this into practice

The lowest-friction test takes ten minutes and does not require trusting the benchmark.

Run it on a PR you already reviewed. Install, configure a provider, and point ocr review --from main --to <branch> at a branch a human reviewed last week. Compare the findings to the human review. You are looking for two things: did it point at the right lines (position drift is the failure the tool claims to fix), and did it miss anything the human caught (recall is the trade it admits).

Try delegation mode inside Claude Code before you buy an API key. ocr delegate preview shows you the file selection and rules it would apply. If the selection matches what you would have asked a reviewer to look at, the deterministic half is doing its job, and you can let your existing agent do the rest.

Use ocr scan on the directory nobody understands. Every codebase has one. A whole-file scan with the built-in rules is a cheap first audit, and the JSON output (--format json --output) is something you can diff against a later scan.

Write one custom rule before you write ten. The rules documentation covers path filtering and targeting. Pick the one class of bug your team keeps merging, encode it, and check whether the template matcher fires on the right files. If it does, you have moved a review decision out of the prompt and into something you can test.

Wire the JSON into CI last, not first. The tool supports it, but a reviewer that comments on every PR with lower recall than your senior engineers will train people to ignore it unless the precision is as high as claimed. Prove that on a handful of PRs before it becomes a gate.

Where this breaks

The benchmark is Alibaba's own. AACR-Bench is described as 50 popular open-source repositories, 200 real pull requests, 10 languages, and 1,505 annotated ground-truth issues cross-validated by more than 80 senior engineers, and the dataset is published on Hugging Face. That is more transparent than most vendor benchmarks, and it is still the project measuring itself against a competitor with a dataset the project built. Reproduce the one comparison you care about before you cite the precision number to anyone.

Lower recall is a real cost, not a footnote. The README frames it as precision over noise, and for a busy team drowning in false positives that is the right trade. For a team that wants a second pair of eyes to catch what the first missed, it is the wrong one, and no amount of deterministic scaffolding changes that.

The origin story is a claim. "Served tens of thousands of developers and identified millions of code defects" is the README's description of its own history at Alibaba, and there is no way to check it from outside.

The rule examples skew toward server-side classes of bugs. NPE and thread-safety rules tell you where the tool grew up. Whether the multi-language ruleset is as sharp on a TypeScript frontend or a Python data pipeline is something you will have to find out on your own code.

Deterministic file selection can filter out something you wanted reviewed. That is the flip side of "no important change is missed": the definition of important is in the tool's code, not yours, until you write rules. ocr delegate preview exists so you can see the selection before trusting it.

And the license has a small inconsistency: the LICENSE file's copyright line reads "Copyright 2026 alibaba/open-code-review Contributors" while the README's footer reads "Copyright 2026 Alibaba." Both are Apache-2.0, so it changes nothing about what you can do with the code, but it is the kind of drift a tool built around precise positioning should probably fix.

The decision the repo makes for you

Open Code Review is a bet that reliability comes from removing decisions from the model, and it ships with the evidence you need to judge the bet on your own terms: a published dataset, an honest recall admission, and a delegation mode that lets you keep your agent and take only the scaffolding.

You can install it and review one PR this afternoon. You can steal the architecture without installing anything: pick the steps in your own agent that must not go wrong, and ask whether any of them should still be a prompt. Or you can wait for the general-purpose agents to stop cutting corners on big diffs. Two of those options are things you control.

Sources: alibaba/open-code-review on GitHub (README, LICENSE, v1.12.4 release); AACR-Bench on Hugging Face; Trendshift daily board, September 16, 2026; IBM Research on the consistency gap (September 15, 2026).

Medium metadata

Title: Open Code Review: Alibaba Put the Parts That Must Not Fail in Code, Not in the Prompt
Subtitle: Alibaba open-sourced the AI code reviewer it ran internally for two years. The interesting part is everything it refuses to let the model decide.
Tags: Code Review, AI Agents, Claude Code, Open Source, Software Engineering
SEO description: alibaba/open-code-review hit #2 on Trendshift with a hybrid design: deterministic file selection, bundling, and rule matching around an LLM that only writes comments. What it does, how to run it with Claude Code or Codex, and where its self-run benchmark and lower recall should give you pause.
Canonical: fervorai.dev (import to Medium from the published URL)