Alibaba's open-code-review Argues Your Review Skill Is the Problem, Then Ships as a Skill Anyway

Alibaba open sourced the AI code reviewer it ran internally for two years. The README opens by listing everything wrong with doing code review through an agent skill, and the fix is a Go binary that the skill then calls.

Most repositories that trend on the strength of an agent skill are selling you the skill. This one opens by telling you why skills do not work for the job it does.

alibaba/open-code-review landed on the trending board fourteenth this week, and its README has a section titled "The Problem with General-Purpose Agents" that reads like a bug report filed against the entire format. If you have used Claude Code with Skills for code review, it says, you have hit three things. Agents cut corners on larger changesets, reviewing some files and skipping others. Reported issues land at the wrong line numbers. And quality swings on prompt wording, because natural-language skills are hard to debug.

Then it names the cause in one sentence: "a purely language-driven architecture lacks hard constraints on the review process."

I have watched all three of those failures happen and never seen a vendor put them in the first screen of their own README. That candor is the reason this repo is worth an hour of your week even if you never run it.

What the split actually looks like

The design is a hard line between what compiled code decides and what the model decides.

On the deterministic side, written in Go and driven by Git diffs: file selection (exactly which files get reviewed and which get filtered, so nothing important is skipped), file bundling (related files grouped into one review unit, so message_en.properties and message_zh.properties travel together), rule matching through a template engine rather than through prompt text, and two separate modules that exist only to fix the model's output, one for comment positioning and one for comment reflection.

That bundling detail is the clever part. Each bundle runs as a sub-agent with isolated context. It is divide-and-conquer, and it is why the tool claims stability on very large changesets where a single agent with one context window starts triaging silently. Concurrency comes free from the same structure: the default is eight file workers, tunable with --concurrency when you hit rate limits.

On the model side, two things only: judgment calls and context retrieval. The agent can read full files, search the codebase, and inspect other changed files to understand a diff. Its prompts and its toolset were both tuned specifically for review, and the README says the toolset was distilled from analysis of production tool-call traces, including call frequency distributions and per-tool repetition rates. That is an unusual thing to build. Most agent tools ship whatever toolkit the framework came with.

The rules layer is a four-tier chain, first-match-wins: the --rule flag, then <repo>/.opencodereview/rule.json, then ~/.opencodereview/rule.json, then an embedded default ruleset covering NPE, thread safety, XSS, and SQL injection. Project rules are a committed JSON file with glob paths, which means your team's review standards live in version control next to the code they govern instead of inside somebody's prompt.

And then it ships as a skill

Here is the turn that makes this repo interesting rather than just useful.

After arguing that skills are unreliable for review, Alibaba publishes a skill. npx skills add alibaba/open-code-review --skill open-code-review installs it. There is a Claude Code plugin too, and a raw command file you can curl into .claude/commands/.

Read the SKILL.md and the contradiction resolves. The skill does not perform a review. It checks that the ocr binary is installed, checks that an LLM is configured, gathers business context to pass through --background, shells out to ocr review --audience agent, then sorts the returned comments into High, Medium, and Low buckets and decides whether it is allowed to apply fixes.

That is a launcher. All the behavior that must not go wrong sits in the compiled binary. The markdown holds only the parts where being approximately right is acceptable: how to phrase the invocation, how to triage output, when to ask permission before editing files.

I think that is the actual lesson here, and it generalizes past code review. The interesting question for anyone writing skills right now is not "what can I express in markdown," it is "which half of this job tolerates a bad day from a language model." Everything on the intolerant side belongs in a binary with tests. The skill is how the agent finds it.

The parts of the SKILL.md that tell on the design

Ship a skill for your own tool and you end up documenting your own failure modes, because the agent needs to handle them. This one is honest to a fault, and the Gotchas section is the most informative thing in the repository.

Comments come back with start_line and end_line, and the file states plainly that "both 0 means positioning failed." The recovery procedure runs four steps: read the comment, open the file, work out from context where the issue actually is, then apply the fix there. So the tool whose headline feature is precise line-level comments ships with a documented path for when the line is missing, and hands that path to the model it just finished describing as unreliable at positioning.

Two more numbers worth knowing before you run it. The default MAX_TOKENS is 58,888 per request, and files with very large diffs may be truncated. And any diff over 50 changed lines triggers an extra risk-analysis phase before the main review, which the file says adds latency but improves quality. Neither of those appears in the README.

The Low bucket is the one I would watch. The skill instructs the agent to discard Low-priority comments silently, defining Low as "likely false positives, lacking sufficient context, nitpicks, or meaningless suggestions." That is a reasonable default for signal-to-noise and a genuinely bad default for your first week, when you have no idea whether the Low bucket is noise or is your codebase's real problems being misclassified by a ruleset tuned on somebody else's stack.

Put this into practice

The lowest-friction version takes about twenty minutes and does not touch CI.

Install the CLI with npm install -g @alibaba-group/open-code-review, then run ocr llm test to confirm your endpoint works. It reads Anthropic and OpenAI-compatible endpoints and will pick up ANTHROPIC_BASE_URL and friends if you already have them exported.

Set language to English first. The config default is Chinese, and if you skip this you will get a review you cannot read and conclude the tool is broken.

Now run ocr review --preview on a branch you know well. Preview makes zero LLM calls and just prints which files it selected. Spend real time on that list, because file selection is the deterministic layer's headline claim and it is the one thing you can grade for free. If the selection is wrong for your repo, no amount of model quality rescues the run.

Then do a real review on a pull request you already reviewed by hand, and grade three things separately: did it find what you found, did the line numbers land, and how many comments arrived. Compare against whatever review skill you are running today on the same diff. That comparison is the whole point, and it takes one afternoon.

If it earns a place, write a .opencodereview/rule.json in the repo before you wire up CI. Two or three rules that encode standards your team actually argues about in review will do more for the signal than any model upgrade. Use ocr rules check <path> to confirm a rule matches the files you think it does.

For CI, the machine-readable path is ocr review --from origin/main --to origin/branch --format json --audience agent, and there are GitHub Actions and GitLab CI examples in the repo.

Honest limitations

I have not run this against a codebase and graded the output, so treat everything above as a reading of the source and the docs rather than a benchmark. The claims I cannot check are the big ones: two years of internal use, tens of thousands of developers, millions of defects found. Those are Alibaba's numbers about Alibaba's private deployment, unverifiable by anyone outside the company, and they are doing a lot of persuasive work in the README.

The default ruleset is the clearest fit problem. NPE, thread safety, XSS, and SQL injection is a Java and web-service failure taxonomy, and the rule examples in the docs are .java files and MyBatis mapper XML. A Rust or Go shop should expect to write most of its rules before the signal beats the noise, and should not read a quiet first run as a clean codebase.

Star counts on this repo are moving fast enough to be worth a caveat. The GitHub page served 902 stars and 51 forks on one load this afternoon and 1k stars and 71 forks on a load a few minutes later, in the same session. Treat any number you see today, including mine, as a snapshot of a curve rather than a measurement.

One thing I will not hedge: telemetry is off by default, and the telemetry.content_logging option that includes prompts and responses in exported data is a separate opt-in. That is the right way around, and plenty of tools in this category do not do it.

What to do with this

The format argument this repo is making will outlive this repo. Skills won distribution because a folder with a SKILL.md is the right size to share. That does not make markdown the right place to put the steps that must not fail.

So take the split, not the tool. Look at whatever agent skill your team leans on hardest, and mark every step where a bad day from the model produces a wrong answer that looks right. Those are the steps that want deterministic code and a test suite. The skill's job is to know when to call it.

Then go run ocr review --preview on your own repo and see whether it picks the files you would have picked. That takes five minutes and tells you more than this article did.

Sources: alibaba/open-code-review README; open-code-review SKILL.md; repository home and live star/fork counts; npm package @alibaba-group/open-code-review.