WASTE Keeps a File of Everything It Got Wrong. Read docs/LEARNED.md Before You Read the Benchmark.
The C engine that runs Kimi K3's 2.78 trillion parameters on a 64 GB laptop publishes 31 dated sections of refuted hypotheses. In a category built on headline memory numbers, that file is the only claim an outsider can check.
Halfway down WASTE's README, past the part where a 2.78-trillion-parameter model runs on a MacBook, sits a sentence most projects would never print. The ideas, hypotheses, priorities, tests, and decisions are human. The code is written by LLMs.
Then, in the project-status section, an instruction to anyone who wants to contribute: read docs/LEARNED.md before proposing an optimization, because failed ideas and negative results are kept there deliberately.
That file runs 31 numbered, dated sections. Most of it is things the team believed and then disproved, kept in place with the wrong version intact. The header states the rule plainly: where a belief turned out wrong, the wrong version stays, because "the refutations were worth more than the confirmations."
I think that file, and not the tokens-per-second number, is the reason to pay attention to this project.
The category has a credibility problem, and the fix is boring
Three separate projects hit the trending boards this week making versions of the same claim: a model far bigger than your RAM will run on your machine. WASTE puts the full Kimi K3 on a 64 GB MacBook Pro. kimi-k3-in-c puts the same model in 8.24 GB. AirLLM fits a 70B on a 4 GB GPU.
The top reply on AirLLM's Hacker News thread said what a lot of people were thinking: projects like this keep showing up, most seem vibe coded, and probably will not be maintained.
That skepticism is correct and it is also useless on its own, because it applies evenly to a category where the projects are not even. You need a filter that separates them. Star counts do not do it. Neither does the headline number, because the headline number is always the flattering one.
Here is the filter I have settled on. A project that publishes what it disproved is making a claim you can audit. A project that publishes only its best result is quoting a ceiling.
What is actually in the file
docs/LEARNED.md is append-only. Sections carry dates and never get rewritten, so the same measurement appears more than once as the engine changed underneath it, and the file tells you how to resolve the conflict: later wins, and take end-to-end numbers from the README.
Section 8 is the clearest example of why that discipline matters. Before the K3 weights finished downloading, the team composed a projection out of already-measured pieces: 954 GB of experts on the internal SSD, 17.1 GB read per cold token, a 46 GB cache clearing one token's working set 2.6 times, roughly 40% hit rate, about 0.8 seconds of I/O per token. The projection landed at roughly 1 token per second.
The measurement came in at 0.45 to 0.62. The projection is still in the file, wrong, with the reasoning attached.
Section 7 documents two hypotheses that both looked right. The first was index locality: blocking the layout so a tile's rows sit contiguous measured a 1.44x speedup in isolation and changed nothing at all in the real engine, because the microbenchmark did not model twelve threads sharing L2 cache. Testing it cost a full reconversion of a 19 GB container. The second was table bandwidth: re-reading an 884 KB table per 64-row tile worked out to 165 GB/s, suspiciously close to the exact memory ceiling of the test machine. Cutting that traffic made the engine slower. The table was shared, read-only, and already cached.
The method note at the end of section 9 is the one I have quoted to other people since: numbers that look too neat deserve suspicion. "Exactly the machine's ceiling" was a coincidence, not an explanation.
Section 3 kills the project's own original design. Shared low-rank basis vectors across experts were the centerpiece of the format, and at matched bit budgets the scheme lost badly, 28.9% weight error against plain per-row INT4's 15.2%. The structural reason got measured separately: Kimi's experts turn out to be close to mutually orthogonal, with pairwise overlap of their dominant subspaces at 0.046 against a random baseline of 0.031. There is nothing to share. The feature is parked rather than deleted, with the criterion for reviving it written into the format doc.
The cache table in the README is the most practically useful negative result, and it will save somebody a weekend:
| expert cache | hit rate | decode |
|---|---|---|
| 3.32 GB | 29.1% | 0.56 to 0.58 tok/s |
| 17.32 GB | 36.2% | 0.63 tok/s |
| 23.32 GB | 38.4% | 0.07 to 0.09 tok/s |
| 29.32 GB | 41.3% | 0.07 to 0.08 tok/s |
Give the process more memory and the hit rate keeps climbing while throughput falls by a factor of eight. The engine stays inside its budget; the machine does not, so every cache hit turns into a page fault. That is the kind of result you only find by measuring past the point where the graph was supposed to keep going up.
Section 27 is my favorite, and it is not about inference at all. The project's CI cross-compiled for Windows for a year and never once built or ran the test suite on Windows. An outside contributor's first native run surfaced four defects, none of them in the engine, all of them in the parts CI could not reach. The line the team wrote afterward is worth stealing: a job that agrees with you is worse than no job.
Put this into practice
Do not start with K3. Build the engine and run the model-free test suite first. git clone, make, make check needs no weights and no network, and it builds a small synthetic model to check itself. Then run Kimi-Linear 48B: a 19 GB container, a 1.28 GB memory floor, about 10.65 tok/s on the same hardware, with an OpenAI-compatible server, tools, structured output, and images. That is a working local model in an afternoon.
Benchmark your own storage before you buy any. Section 2 measured an external USB SSD at 0.94 GB/s against an internal SSD at 12.78 GB/s, using the engine's real access pattern (12 MB random reads, F_NOCACHE, pread, N threads) rather than a sequential dd. The enclosure was the ceiling, not the NVMe inside it, and it did not scale with threads. If you are planning any streaming-inference work, measure the pattern you will actually use.
Leave --budget alone. The engine picks a safe budget, reports it, and refuses to start below the floor. The cache table above is why overriding it is usually a mistake.
Then steal the format for your own repo. A useful entry needs five things: the date, the hypothesis in one sentence, the measurement with the machine attached, the verdict, and what testing it cost. The reconversion of 19 GB is part of the record. That last field is what turns the file from a diary into a decision aid, because the next person can weigh whether the experiment is worth repeating.
Where this gets thin
Everything in the file was measured on one machine, a 64 GB MacBook Pro with an M5 Pro, and the file says so at the top. That is honest and it is still a sample size of one. Your NVMe, your thermals, and your core count will move these numbers.
The repository is moving fast and says so: the format and API are not frozen, a large QA run happens before each release, and the maintainers write that instabilities are definitely possible. At the time of writing it sits at 807 stars, 79 forks, and 153 commits, which is a young project by any measure.
A negative-results file is still self-reported. Nobody audits what did not make it in. The discipline is real and it is not the same thing as external review.
It also proves less than it looks like it proves. LEARNED.md tells you the hypotheses were tested. It does not tell you the code is correct. That claim rests somewhere else, on the PyTorch oracle comparisons: max absolute logit difference of 4.2e-05, identical argmax and top ten, layer-by-layer checks, and a vision tower matching its own oracle to 2.3e-06. Read those separately and judge them separately.
And the K3 result is a measurement rather than a workflow. Running it means roughly 1 TB of internal NVMe for the container, another 1.42 TB of temporary staging if you convert the weights yourself, about 4.7 hours of conversion, and then 0.45 to 0.62 tokens per second. Everyone quoting the memory number should also quote that one. For comparison, kimi-k3-in-c reports about 33 seconds per token at its 8.24 GB minimum preset and about 20 seconds at a 128 GB budget, and AirLLM's own notes put K3 on a 48 GB RTX 6000 Ada at 292 seconds per token. None of those are lies. They are just different points on the same curve, and the curve is the thing that goes missing when a project only publishes its best number.
What to do with this
Pick one repository you are currently trusting on a benchmark and go looking for its negative results. Not its issues, not its roadmap. A place where somebody wrote down a thing they expected to work, measured it, and reported that it did not.
If you find that file, you have learned something real about the project. If you do not, you have learned something too, and it costs you nothing to keep looking before you spend a terabyte of disk finding out yourself.
Then write one for whatever you are building. The first entry is the hardest, because you have to admit the thing in your head was wrong. Every entry after that is just bookkeeping, and it is the cheapest credibility any project can buy.
Sources: sqliteai/waste README and docs/LEARNED.md, FareedKhan-dev/kimi-k3-in-c, lyogavin/airllm, Hacker News discussion.