Ship an Attestation, Not a Benchmark: What litelm's README Does That a Pass Rate Cannot

A 2,900-line rewrite of a 100,000-line library published its review boundary instead of a score, and the pattern is free to copy in any README you maintain

Most READMEs for a reimplementation lead with a number. Faster, smaller, higher on some chart. A project called litelm, which reached v0.5.2 on September 11, leads with a boundary instead. Under a heading called "Upstream attestation," it names the exact commit range of the upstream library its author reviewed, 649eb2d through 9a715df2, says he triaged 360 core-path commits inside it, gives the test result, 262 passing and 55 skipped, and then does the thing almost nobody does. It states what the attestation does not cover:

This attests litelm's declared routing/formatting/DSPy surface only, not full litellm compatibility.

Read that twice, because it is unusual enough to be worth dwelling on. A maintainer asking you to replace a dependency, in the same paragraph, telling you the limit of what he is vouching for. No benchmark in the repository. No chart. A scope line.

That is a better trust artifact than a score, and the reason why is worth working out, because the pattern costs nothing and almost nothing else in agent tooling is doing it.

The thing litelm is

Briefly, because the mechanism matters more than the product. LiteLLM is the Python library a very large number of teams use to call many model providers through one interface, and to translate between their message formats. litelm extracts the call path and discards everything else: model routing, message translation, streaming, tool use, embeddings, text completions and the OpenAI Responses API stay; the Router, the proxy server, caching, budgeting, cost tracking, token counting, image and audio endpoints, fine-tuning, agents, guardrails and the scheduler are gone. The README's framing is that the core is "buried under 100k+ LOC of proxy servers, caching layers, cost tracking, and dozens of features most users never touch," and what remains is about 2,900 lines with two dependencies, openai and httpx.

The API deliberately mirrors the original, same function names, same arguments, same response types, so the migration is s/litellm/litelm/ across your imports. That choice is what makes the next part necessary, and it is where most projects in this position would have reached for a benchmark.

Why a pass rate would have told you nothing here

Suppose litelm had published a score. "97% compatible." What would you do with it?

You would immediately need to know which 3%. A compatibility percentage is an average over a test population you did not choose and cannot see, and the only question you actually have is whether the specific provider, the specific parameter and the specific response shape your code depends on are in the passing set or the failing set. An aggregate cannot answer that. Worse, an aggregate invites the reader to round up, because 97 reads as "basically yes" when the 3% might be the one streaming path your product runs on.

The attestation answers a different and more useful question: how far did a human actually look. A commit range is checkable. You can open 649eb2d, open 9a715df2, and see for yourself what sits between them, including whether anything landed upstream after 9a715df2 that nobody has reviewed. A count of 360 triaged core-path commits tells you the granularity of the work. A test count of 262 passing and 55 skipped tells you the skipped ones exist, which a percentage would have silently absorbed. And the scope line tells you the author knows his claim has edges.

The difference is falsifiability. A score is a summary you must trust. An attestation is a set of claims a reader can go check, and which would be embarrassing if checked and found false. That asymmetry is the whole value.

The mechanism under the claim

The attestation is not just prose. litelm ships the machinery that produces it, which is what separates this from a README that merely sounds careful.

There is a script, scripts/sync_litellm_tests.sh, that pulls the upstream project's own tests into tests/ported/. There is a second one, scripts/ported_contract.sh, described in its own comments as a "fast upstream compatibility gate for litelm's explicit public contract." It refuses to run if the ported tests are absent, tells you to run the sync script first, then selects a node list from scripts/ported_contract.py and runs exactly those against litelm:

bash scripts/sync_litellm_tests.sh    # pull upstream's tests
bash scripts/ported_contract.sh       # run the public-contract subset

So the compatibility claim is defined as "a named subset of the upstream project's own tests, which you can re-run," rather than as a number the maintainer calculated. That is a meaningfully different object. It also means the scope line is not modesty, it is a description of the node list.

The provider table does the same job in a cruder way. Nineteen providers are listed; exactly seven carry "Verified: Yes," and the other twelve say "No." Most projects would have written "19 providers supported." This one tells you that twelve of them are untested routing paths, which is the single most decision-relevant fact in the document.

What it still does not tell you

The honest read is that litelm's attestation is good practice and weak evidence, and the two are not the same thing.

Everything in it is self-run. The tests are the maintainer's, the triage judgment is the maintainer's, the review of what counts as a core-path commit is the maintainer's, and nobody independent has confirmed any of it. The project calls itself Alpha in its own Status section. A reader who trusts the attestation is still trusting one person, just trusting them about something specific and checkable rather than about a number.

The README is also unusually plain about its own provenance, in a section called "Development transparency": much of the code was written with Claude Code using Claude Opus 4.6 and 4.7, and code from 2026-05-14 onward was written through Pi using GPT-5.5, with compatibility claims resting on "tests and maintainer review, not AI authorship." That disclosure belongs in the same family as the attestation. It tells you where to aim your skepticism.

And the structural limit does not go away: litelm measures its own compatibility. A project that both performs a transformation and reports on the transformation cannot see the failure modes its own framing excludes. The attestation narrows that blind spot by naming a boundary. It does not remove it.

Put this into practice

The pattern generalises to anything you maintain where someone has to decide whether to depend on you. A skill, an MCP server, a fork, a harness extension, a client library, an internal package another team consumes. Four moves, none of which need a benchmark harness.

Name the range you reviewed. If your project tracks an upstream, put the two commit SHAs in the README. "Reviewed upstream from X through Y" is a fact with a date attached, and it tells a reader exactly how stale your compatibility claim is without you having to maintain a freshness badge. If you do not track an upstream, name the version or spec revision you built against.

Write the scope line before you write the pitch. One sentence: this attests A, B and C, and not D. Draft it first, because writing it will change what you claim in the paragraphs above it. If you cannot finish the sentence, you do not yet know what you are promising.

Report counts, not percentages. "262 passed, 55 skipped" survives contact with a skeptical reader. "97% passing" does not, because the first question is always what the denominator was and the second is what got skipped. Counts answer both.

Mark per-item verification status honestly. litelm's provider table with seven yeses and twelve nos is the most useful thing in the document. If you support twenty integrations and have actually exercised six, say six. You lose nothing real, because the adopter whose provider is in the untested twelve was going to find out either way, and they will find out in production rather than in your README.

Make the gate runnable. A claim a reader can re-execute is worth more than the same claim asserted. Ship the script. Even a crude one that pulls upstream tests and runs a named subset turns your attestation from a statement into a procedure.

A cheap version of all of this fits in twelve lines:

## Attestation (2026-09-12)
Reviewed upstream `abc1234`..`def5678` (142 commits on the paths we wrap).
Tests: 118 passed, 21 skipped (`make contract`).
Verified end to end: Postgres, SQLite. Not verified: MySQL, MSSQL, Oracle.
Attests the documented query and migration surface only, not full ORM parity.
Self-run; no third-party review.

That block is more informative than any score you could put in its place, and you can write it this afternoon.

Honest limitations

I did not run litelm, did not execute its test suite, and did not verify that the ported-contract gate does what its comments say. I read the two scripts and the README; the behaviour described is the behaviour the code claims, and I confirmed the files exist and are fetchable at the current main.

I also did not independently verify the "100k+ LOC" characterisation of LiteLLM. That figure is litelm's own framing of the upstream project, and my attempt to pull LiteLLM's package metadata for context returned an empty response, so the size contrast in this piece rests on litelm's claim rather than on my measurement. The same applies to the "~2,900 lines" figure for litelm itself, which is the README's number.

Nothing here is a recommendation to swap a production gateway. The project is self-declared Alpha, seven of nineteen providers carry a verified mark, and there is no router, no fallback and no cost tracking, so for most teams litelm is a diagnostic for discovering which LiteLLM features they actually depend on, not a replacement. Its Hacker News reception is modest and was still moving while I wrote this: the live item read 109 points and 40 comments at 15:40 ET and 169 points with 61 comments about forty minutes later. Quote a figure for a rising story and you are quoting a timestamp, not a fact.

Finally, the argument in this piece is about what a README should contain, and I am making it from one good example. A pattern with one instance is a suggestion, not a finding. If attestations turn out to be worse than scores at scale, the reason will probably be that maintainers write them once and never update the commit range, which is a failure mode a score does not have.

What to take from it

Benchmarks answer "how good is this." Adoption decisions mostly turn on "what exactly are you promising me, and how would I know if you were wrong." Those are different questions, and the second one does not need a number.

You cannot buy an independent audit of your side project. You can publish the boundary of your own diligence, in a form a stranger can check in ten minutes, which is the part of an audit that does the work.

Open the README of the thing you maintain. Find the sentence that makes a compatibility or capability claim. Write the scope line that goes underneath it. If that sentence is hard to write, you have just learned something more useful than any benchmark was going to tell you.

Sources: kennethwolters/litelm · litelm on PyPI · HN discussion

Medium metadata

Title: Ship an Attestation, Not a Benchmark: What litelm's README Does That a Pass Rate Cannot
Subtitle: A 2,900-line rewrite of a 100,000-line library published its review boundary instead of a score, and the pattern is free to copy in any README you maintain
Tags: open source, software engineering, llm, developer tools, documentation
Canonical: import from the fervorai.dev URL
Kicker image suggestion: two README blocks side by side, one showing a large percentage, one showing a commit range and a scope line