OpenWiki 0.4.0 Proves Its Claims Against Your Code. The Claims It Can't Pin Look Exactly the Same
LangChain's documentation CLI now stores every factual claim with a pointer to the exact lines that justify it, then re-checks all of them on each update with zero model calls. The verification mechanism is better engineered than the headline number suggests, and its coverage gap is invisible to the agent reading the page.
The interesting decision in OpenWiki 0.4.0 is one that never made it into the announcement post. When the tool pins a claim to a range of code, it does not store a git commit SHA. It stores a content hash plus what the codebase calls relocation anchors, the first and last selected lines and their surrounding context.
Why that matters: add an import at the top of scheduler.ts and every line number below it shifts. A SHA-based system flags the whole file. A line-number system flags every claim beneath the edit. OpenWiki relocates the same selected text and leaves the version token unchanged, because "versions are content-derived, a version mismatch is exactly a content change" (from the project's own generated wiki at openwiki/concepts/grounded-claims.md).
That's a careful piece of engineering solving a real problem. And it applies only to the claims OpenWiki was able to attach to a repository file in the first place. Everything else in your wiki, the design rationale, the decision someone made in a thread, the constraint that came from an external doc, is not a weakly-verified claim. It is not a claim at all.
The number to ask for is coverage, not accuracy
LangChain's post leads with 97.8% supported claims. Good number. Here's the question it doesn't answer: what percentage of the sentences in your wiki became claims?
Nothing published tells you. The README draws the boundary explicitly: "Grounded Claims currently apply to repository code wikis and repository evidence. Connector-derived facts, including LangSmith-only observations, are not claimed." Also excluded from grounding: index.md, log.md, instructions.md, anything under .claims/, and any evidence pointing at git metadata or at OpenWiki's own generated output.
So the self-correction guarantee covers a subset. My position is that this is fine, and that the way it's presented is the problem. A tool whose entire pitch is "your agent docs go stale and lie to your agent" should be loudest about the region where it still can't help, not quietest.
Because here's the part that turns a scoping decision into a practical hazard.
The Markdown gives the agent nothing
Claim state is not stored in the prose. The README: "The Markdown stays clean; structured Claim state lives alongside it under openwiki/.claims/." The concepts page says the same thing more bluntly: "Claim state is not stored inside the generated Markdown."
The only thing visible in the file itself is page-level front matter, a sources list and a verified: {by, at} stamp, and the page earns that stamp only after a successful submission reconciles a complete claims set and passes a final evidence recheck.
Sit with what that means for the consumer. Your coding agent opens a wiki page. Sentence four is a claim that was re-hashed against src/scheduler.ts#L393-L404 eleven minutes ago. Sentence five is a design rationale nobody could pin to any file, so it was never claimed, never checked, and possibly written by a model four months ago that inferred it. In the rendered Markdown those two sentences are typographically identical. There is no per-claim marker of any kind. Reading the sidecar is a separate act that nothing in the page prompts.
Page-level verification is a real signal. It is a coarse one, and it is being asked to carry sentence-level trust.
How the mechanism works
Worth understanding properly, because the design is good.
A claim is a statement plus an evidence array. The blog's own example:
{
"statement": "Failed tasks are retried three times by default.",
"evidence": ["repo://src/scheduler.ts#L393-L404"]
}
Each resolved resource gets an opaque version token. Whole-file evidence is versioned as repo-file-v1:sha256:.... Line-range evidence is versioned as repo-lines-v1:sha256:..., hashing the selected content plus those relocation anchors.
The check runs at the start of every wiki update, before the agent does anything, and it is deterministic. LangChain's wording: "This is a deterministic check with no model calls, so it stays fast even as the wiki accumulates thousands of claims." The consequence is the good kind of boring. Update cost scales with how much code changed, not with how many claims you hold. A wiki with 40,000 claims and one changed file is a cheap update.
The runtime marks a claim unresolved when an evidence resource no longer resolves at all, and stale when everything resolves but at least one resource came back at a different version. Unresolved wins over stale. The agent never sweeps the set itself; stale claims surface to it when it reads that page, and anything unresolved stays flagged for a future update instead of being dropped.
The evaluation numbers, assigned correctly, because they get muddled in secondary coverage. The main table compares the same replay run with and without the claims runtime, n = 2000 per arm:
| Baseline | Claims | |
|---|---|---|
| Supported | 92.9% | 97.8% |
| Stale | 3.5% | 0.5% |
| Hallucinated | 0.7% | 0% |
| Unverified | 2.9% | 1.7% |
In counts: 80 stale claims down to 9, 15 hallucinated down to 0.
The dramatic 77% to 98% recovery you may have seen elsewhere is a different table, a single run across two checkpoints where a code change left 17% of that run's claims stale at T1 and the next checkpoint brought it back to 98% supported with 0% stale. Both tables are real. They measure different things.
Put this into practice
Upgrading is one line, and the migration is automatic:
npm install -g openwiki@latest
You need Node 22 or newer. On your next openwiki --update, 0.4.0 starts generating claims and migrates your wiki to Open Knowledge Format v0.2, which adds generated: {by, at} provenance to new pages and projects claim evidence into a sources list.
Turn telemetry off first if that matters to you, because it ships on. Set OPENWIKI_TELEMETRY_DISABLED=1 or DO_NOT_TRACK=1, or add the first one to ~/.openwiki/.env to make it permanent. In CI, put it in the workflow environment. There's a --telemetry-file=<path> flag if you want to see the exact payload before deciding.
The habit worth building alongside the upgrade costs nothing and pays immediately: separate your pinnable facts from your unpinnable ones by page. Put behavior, defaults, and API shapes on pages that can earn a verified stamp. Put architecture rationale, historical decisions, and anything sourced from a conversation on their own pages, and title them so an agent reading the index knows it's holding unverified context. OpenWiki won't do this split for you. Doing it by hand restores the signal the clean Markdown removed.
If you want to check the claims about the claims, the harness is checked into the repo at evals/ledger/ with a documented pnpm run eval:ledger entry point and two benchmarks, calc and taskflow, in the tree. Genuine credit for that. Most vendors publish the score and keep the scorer.
Honest limitations
The blog post has no limitations section. None. The closest it comes is a design note that a stale claim isn't necessarily wrong, only no longer safe to assume. The real caveats are in the repo, which is where I got these.
The benchmark's headline metric is blind to omission. From evals/ledger/README.md: "The score does not measure whether the wiki covers every important source topic; that limitation remains explicit." A wiki that says very little, accurately, scores beautifully. Worth knowing: the benchmark's earlier draft, submitted as KEB (Knowledge Evolution Benchmark) in PR #622, used a harmonic mean of coverage and precision specifically because, in its own words, "the harmonic mean prevents systems from scoring well by saying almost nothing." Coverage got dropped from the shipped headline score. That's a regression in the metric's honesty between draft and release, and it's visible in the repo's own history.
The evaluator is a language model. LEDGER's grounding judgments run through bounded, schema-validated evaluator calls with BM25 retrieval and a human-reviewed evidence map for routing. The headline percentages are model-judged, not ground-truth-labeled. The README is upfront about this and reports a separate evaluator completeness rate. The unverified bucket is partly an evaluator-uncertainty bucket by design, catching dropped qualifiers and misread diagrams so they don't become false stale results.
Self-designed, self-run, self-reported. GitHub user @colifran authored PR #638, the grounded claims feature, and PR #622, the benchmark that measures it, and the blog post announcing the result. The harness being open source is a real mitigant. What isn't published is which repository produced the n=2000 numbers or under what configuration. The checked-in fixtures show 35 to 50 claims per checkpoint, so reaching 2000 implies many checkpoints or aggregated runs. Nothing published shows this run against a large real-world open-source codebase.
The docs are behind the code. As of today, docs.langchain.com/oss/openwiki/overview and the code-mode page still say okf_version: "0.1" while the README and shipped code are on v0.2, and there is no grounded-claims page in the docs section at all. A tool built to stop documentation drift shipping with drifted documentation is not a scandal. It is a useful reminder about what this class of problem is like.
Project maturity. MIT, TypeScript, roughly 15.6k stars, and 64 open pull requests against 61 open issues on a project that was on 0.3.x eleven days ago. Also note that GitHub's HTML is currently serving wildly different cache generations of this repo's star count. I saw 58, 695, 12.2k, 14.8k, and 15.6k across a single session, ordered by cache age. If you're citing a figure, pull it from img.shields.io/github/stars/langchain-ai/openwiki.json and treat any GitHub page number as suspect.
What to do with this
Grounded claims are the right shape for agent memory, and I'd rather run 0.4.0 than 0.3.x on any repo where agents read the docs. Deterministic verification with no model calls is the sort of design decision that keeps working when your wiki gets large, which is exactly when the old approach fell apart.
Just don't let the 97.8% do work it can't do. Before you trust a page, open openwiki/.claims/ for it and count. Claims present versus paragraphs written is a ratio nobody is reporting, it takes about a minute to compute on your own repo, and it is the number that tells you how much of your agent's context is actually being checked.
Then decide what you want to do about the rest of it, because right now nothing and nobody is watching that part.
Sources: Building Self-Correcting Memory in OpenWiki, langchain-ai/openwiki, v0.4.0 release, OpenWiki docs, telemetry / customize docs, Open Knowledge Format spec.