WASTE Runs Kimi K3's 2.78 Trillion Parameters on a Laptop, and the Bottleneck Moved to Your SSD
A dependency-free C engine puts a trillion-scale frontier model on a 64 GB MacBook. The README is unusually honest about what that costs, and the cost is not what you'd guess.
Somebody ran the full, unmodified Kimi K3 on a laptop last week. Not a distill, not a 2-bit surgery, not a 30B sibling wearing the family name. All 2.78 trillion parameters, from the same 96-shard checkpoint Moonshot published, generating text on a MacBook Pro with 64 GB of unified memory.
It produced about a third of a token per second. Fifty seconds for one sentence.
Both halves of that matter, and the second half is where the actual engineering is. WASTE, released July 30 by SQLite Cloud, is roughly 6,000 lines of C with no third-party runtime dependencies, and its README is the most self-critical repository document I've read this year. It tells you where the demo breaks before you find out yourself.
The idea, and why it works on this class of model
The premise is stated in one line: "almost all of that weight is idle at any instant, and idle weight does not need to be in memory, it needs to be reachable in time."
A mixture-of-experts model activates a small fraction of itself per token. K3 touches 16 experts in each of 92 layers, which the README describes as roughly 4 percent of the model per token. Everybody has known this. The training-cost story got told repeatedly. Almost nobody rebuilt the runtime around it.
WASTE does. Convert the checkpoint once into a .waste container, which is a directory holding a JSON manifest, a resident trunk, and one expert bank per layer. The trunk, quantized at 4 and 8 bits, stays in RAM at 27.28 GB. Experts live on NVMe. Whatever RAM is left becomes a bounded LFRU cache over the banks.
The layout details are where the speed lives. Each expert record is 4 KiB-aligned with its gate, up, and down matrices adjacent, so routing to an expert costs exactly one pread. Not three, not a seek per matrix. Expert weights use residual vector quantization at three bits per weight, three stages of 256-entry codebooks over 8-dimensional vectors, and the matrix is never materialized. Per token the engine builds partial dot-product tables so every expert row is three table reads and two adds.
The engine also bypasses the OS page cache on purpose, with F_NOCACHE on macOS, O_DIRECT on Linux, FILE_FLAG_NO_BUFFERING on Windows. The README's justification is the best sentence in the repository: "a cache you do not control is not a cache, and the corollary is that an engine should stop asking for memory before the OS starts taking it back."
Correctness is checked rather than asserted. Every layer validates against a PyTorch reference. Final logits agree to 3.6e-06. The vision tower matches its own oracle to 2.3e-06.
The numbers, including the ones that hurt
The 1.42 TB published checkpoint converts to a 982 GiB container. The RAM floor is 29.05 GB at 4K context, rising to 30.54 GB at 32K and 35.63 GB at 128K. The engine refuses to start below the floor.
Decode runs at 0.32 to 0.34 tokens per second on an M5 Pro with the container on internal SSD. One sample run: 16 tokens in 49.31 seconds, with experts hitting cache 3,357 times against 20,195 misses, a 14 percent hit rate. Prefill is 0.47 tokens per second chunked. Model load takes 20 seconds.
Each token reads 17.0 GB from disk at about 9.9 GB/s, against that SSD's measured ceiling of 12.78 GB/s. The engine is running the storage bus nearly flat out. Time breakdown at a cold cache: MoE is 82.5 percent of it, split into 53.5 percent expert I/O and 20 percent expert matmul.
So this is not a model that happens to be slow. It's a model whose entire cost is now storage bandwidth, and the compute you paid for is mostly waiting.
Two measurements make that concrete in a way benchmark tables usually don't.
Move the container to a USB enclosure at 0.94 GB/s instead of internal NVMe at 12.78, and the same token takes thirteen seconds. The README says the container "must be on internal NVMe," and that's the reason.
And the memory-budget sweep is genuinely strange. At a 32 GB budget you get a 3.32 GB cache, a 0 percent hit rate, and 0.31 tok/s. At 46 GB you get 17.32 GB of cache, 13 percent hits, and 0.32 tok/s. At 52 GB, throughput collapses to 0.11 to 0.14. At 58 GB it falls to 0.04. Asking for more memory made it eight times slower, because the OS started reclaiming pages the engine needed resident. The usable window on a 64 GB machine is one budget wide. The README's warning to benchmarkers: "The engine is deterministic; the machine is not, and it does not fully recover between runs. Sweep upward."
There is a Metal backend. It's off by default because it is correct and 22 percent slower. Several hundred small dependent matvecs per token is the worst possible shape for a GPU, and the CPU path already saturates memory bandwidth.
Put this into practice
The trillion-parameter demo is not the thing to run first. Run the other one.
The same converter handles Kimi-Linear-48B-A3B-Instruct, and only --src changes. That container is 19 GB, needs 1.86 GB of RAM minimum, and delivers 8.92 tok/s at an 8 GB budget with a 78 percent cache hit rate. That is a usable local model on a machine you already own, and it demonstrates the whole architecture in an afternoon rather than a weekend. Note the tradeoff: Kimi-Linear ships no chat template, so the CLI tells you and runs raw.
If you do want K3, budget realistically. You need about a terabyte free on internal NVMe for the container, plus another 1.42 TB of staging for the download that you free afterward. Conversion took roughly 4.7 hours on an M5 Pro with --jobs 3, or 23.7 hours with the pure-torch encoder. Building the engine needs a C11 compiler and make, with no BLAS, no CUDA, and no Python at run time. Conversion is the only step that wants Python, with torch and safetensors.
Turn on --verify. Expert payload checksums are off by default. The cost is about 1 percent on K3 and 5 percent on Kimi-Linear, which is nothing against silently decoding rotted bytes.
Treat 64 GB as the real minimum for K3 even though the floor says 29. The README is blunt: "A 32 GB machine can technically open the model and will page badly."
If you're embedding rather than experimenting, the public API is 26 functions in src/waste.h, producing one libwaste.a and one waste binary that need nothing at run time beyond libc and pthreads. There's an OpenAI-compatible server in the repo written against the Python standard library, reaching the engine through ctypes.
Honest limitations
The maintainers wrote most of this section for me, which is itself the reason to trust the rest.
The API is not frozen. Chat templates exist only for models whose format has been transcribed from a reference encoder, which today means K3 and nothing else.
The trunk has no checksum at all. Not "off by default," absent. The README is direct about it: "the trunk and the codebooks have nothing to check against in the format, and nothing has been built in its place." The bundled tools/verify_container.py does not fill that gap, since it re-derives against the original weights and answers whether the conversion was correct rather than whether your copy still is.
AVX-512 compiles and dispatches from CPUID and, in the maintainers' own words, "has still never executed an instruction." The development machine is ARM and the hosted x86 runner is an AMD EPYC 7763 that reports no AVX-512.
Windows builds and runs on exactly one toolchain and one CPU, cross-compiled with MinGW-w64. MSVC has not been attempted because the sources use GNU C. ARM64 Windows is not built. And nobody has run a real container on Windows, so the page-cache bypass is unproven there under load.
The novelty claim comes with its own disclaimer, which I've never seen a repository volunteer: the maintainers found no other published demonstration of trillion-scale NVMe streaming on a consumer machine, then wrote that this is "a report of what our search turned up rather than a survey" and invited counter-examples.
One caveat on the numbers you'll see elsewhere. A figure of 0.6 tokens per second is circulating in third-party coverage. It appears in neither the repository nor the author's announcement, both of which say 0.32 to 0.34. The expert count of 896 and the model's release date of July 27 come from Marco Bambini's Substack post only, not the README. The active-parameter count per token is not stated as an absolute number in either source, only as "about 4%."
What actually changed
The headline is that a 2.78-trillion-parameter model ran on a laptop. The finding underneath it is smaller and more useful: for mixture-of-experts models, the binding constraint stopped being RAM and became storage bandwidth and I/O latency.
That reframes a lot of downstream work. Quantization formats designed for resident weights are optimizing the wrong axis. Cache eviction policy is now a first-class performance knob. NVMe write endurance becomes an operating concern for anyone doing sustained local generation. And an agent harness scheduling around 17 GB of disk reads per token is a different scheduler than one scheduling around GPU occupancy.
Bambini's own framing is the right one: at current speed, K3 through WASTE is "primarily a technical result rather than a general-purpose interactive deployment."
Go clone it and run Kimi-Linear tonight on whatever machine you have. Watch the cache hit rate climb past 78 percent and the tokens land at nine per second. That's the version of this idea that's ready. The trillion-parameter run is the version that tells you where the next year of local inference work is going, and it's going to your storage stack.
Sources: sqliteai/waste on GitHub (Apache-2.0, SQLite Cloud, Inc.), README as served; Marco Bambini, "The WASTE inference engine" (July 30, 2026); Show HN discussion. All measurements are the maintainers' own, taken on a 64 GB MacBook Pro M5 Pro with the container on internal SSD.