Independent AI intelligence Two editions daily · ET
Fervor AI
Menu

Analysis · November 8, 2026 · repo

llama-modchipDeepSeek-V4-Flashlocal-aifrontier-modelsclaude-code

llama-modchip Runs a 284B AI Model Without a GPU. The Catch Is the File Format.

A patched llama.cpp fits DeepSeek-V4-Flash into desktop RAM. Here is how it works, what speed to expect, and the lock-in hiding in the files.

For years the price of running a big AI model at home was a graphics card. If the model didn't fit in the card's memory, you didn't run it, or you ran it so slowly that it wasn't worth the electricity.

That rule is starting to bend, and the reason is a design most big open models now share. A project called llama-modchip reached Hacker News on November 8 claiming it runs DeepSeek-V4-Flash, a 284-billion-parameter model, on a 64 GB desktop with zero GPUs. The numbers are real enough to try. The catch sits in the file format, and it matters more than the speed.

The short version: the hardware wall is moving from your graphics card to your system RAM, and llama-modchip shows how far that can go. It also shows the cost of getting there with a fork.

What llama-modchip actually is

llama.cpp is the open-source engine behind most local AI tools; Ollama and LM Studio are friendlier apps built on the same foundations. llama-modchip is a set of patches to llama.cpp, plus its own tools for shrinking models and a small command-line wrapper called ./modchip for chatting, benchmarking and launching a web UI.

The README describes it as CPU-only inference for large "mixture-of-experts" models in 16 to 64 GB of ordinary system memory, on Intel and AMD chips with the AVX-512 or AVX2 instruction sets (most recent desktop processors have at least AVX2) and on ARM64 chips such as Apple Silicon. The license is Apache 2.0. The latest tagged release, v2.4.0, is dated November 4, and the main branch has kept moving since. The maintainer goes by synchopate and gives no other name.

How a 284B model fits in 64 GB

Four tricks stack up. None of them is magic on its own.

1. Most of the model sits idle on any given word. A mixture-of-experts model (MoE) is like a hospital full of specialists: each patient sees a handful of them, not the whole staff. DeepSeek-V4-Flash has 284 billion parameters in total, but the README says only about 13 billion are active for each token it generates. By default modchip runs all six routed experts plus one shared expert per layer, the same routing as the original model. So the model still has to be stored in memory, but each step only does a fraction of the math. That is why a CPU can keep up at all.

2. The file gets squeezed hard. Quantization means storing each number with fewer bits, the way a JPEG stores a photo with less detail. modchip's custom format, which it calls Fractal-64, takes the DeepSeek checkpoint down to a 56.56 GiB file. The README puts it at roughly 58 to 59 GiB of memory in use at a full 256K-token context, on a machine with about 62.5 GiB usable. That is tight. It fits, with almost nothing to spare.

3. The code is written around the CPU's cache. Custom kernels (the low-level math routines) arrange work so the parts used on every step stay in the processor's fast cache, and the cold expert weights don't push them out.

4. A tiny guesser runs ahead of the big model. A small drafter, which the README calls Stockfish-PVS, proposes the next few tokens and the big model checks them in one pass (speculative decoding). When the guesses are right, you get several tokens for the price of one check.

What speed to expect

These are the project's own numbers, measured on the maintainer's 16-core AMD Ryzen 9 9955HX with no GPU, plus rows that community members submitted through the repo's benchmark issue. Nobody has independently audited them.

  • DeepSeek-V4-Flash 284B on a 64 GB machine: about 44 tokens per second reading your prompt, and about 9 to 11 tokens per second writing on the CPU alone. With the drafter, the README reports up to about 14.7.
  • Nemotron-3.5-30B-A3B on a 16 to 32 GB laptop or mini-PC: a 14.35 GiB file, with 25 to 60 tokens per second depending on the task. The fast end is code refactoring, where the drafter guesses well.
  • With a 16 GB AMD graphics card added: about 80.8 tokens per second on the 30B model, with the busy parts on the GPU and the cold experts in RAM.

Ten tokens per second is roughly a slow reading pace. That is fine for chat and for an agent working through a task in the background. It is painful for anything where you sit and wait on long outputs.

The catch: the file only opens in modchip

Here is the line in the README that deserves more attention than the speed table. Fractal-64 files pack their numbers with a custom tensor type (the code that tells software how numbers are stored in the file), ID 40, and the compatibility note says unpatched llama.cpp, Ollama and LM Studio fail on them with unknown tensor type 40.

Think of it as a video saved in a codec only one player supports. The model is DeepSeek's. The file is modchip's. Download the 56 GiB build and you can't move it into the tools you already use. You run modchip's binaries, built from one maintainer's fork, or you download a different file.

That changes what you are adopting. You are not trying a model. You are taking a dependency on a project with one named maintainer and a fast commit pace, whose README has already moved past its last tagged release. If upstream llama.cpp ships a fix you need, you wait for the fork to pick it up.

None of that makes it bad. It makes it an experiment, and you should treat it like one.

Put this into practice

Start small, check your machine, and only then pull the big file.

1. Check your CPU without downloading a model. The README offers a readiness script, scripts/check_cpu_readiness.py, and documents running it by piping it from the internet straight into python3. Don't. Download it, read it, then run it:

curl -sLO https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py
less check_cpu_readiness.py
python3 check_cpu_readiness.py

It reports on your CPU features, 2 MB HugePages and shared memory. ./modchip status covers the same ground once the repo is cloned.

2. Try the 30B model first. It is a 14 GiB download, it runs on a 16 to 32 GB machine, and it tells you whether the stack works on your hardware before you commit 56 GiB of disk (plus room for the download) and an afternoon:

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

3. Move to the 284B build only on a 64 GB Linux box. The README's direct-run example pulls the model from Hugging Face and starts a local server:

hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF --local-dir ./dsv4
chmod +x ./dsv4/llama-server-modchip-linux-x86_64
./dsv4/llama-server-modchip-linux-x86_64 -m ./dsv4/DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf -c 32768 -t 16 -fa on --port 8080

Notice the example uses a 32K context (-c 32768), not the full 256K. Start there. Close your browser tabs before you push the context up, because at 256K the model and its cache eat almost all of a 64 GB machine.

4. Read the launcher before you point a coding agent at it. ./modchip claude launches Claude Code (Anthropic's terminal coding agent, which you need installed separately) wired to the local engine, with saved prompt snapshots in shared memory so the long tool-definition prompt doesn't have to be re-read every session. Open the modchip script first. Its launcher passes --dangerously-skip-permissions, which turns off Claude Code's approval prompts so the agent can run commands and edit files without asking, and it swaps in its own system prompt describing the agent as a "software engineering and security research agent." If you want approvals, start Claude Code yourself and point it at the local server instead of using the wrapper. Expect the 284B model to feel slow as a coding agent either way; the 30B model is the more practical fit.

5. Know the speed dial. The README says setting LLAMA_MOE_TOP_K=5 (or 4) uses fewer experts per layer, trading a little quality for speed. Its table reports about 0.42% higher perplexity (a measure of how surprised the model is by real text) for the release's Top-5 mode. One wrinkle: the tagged v2.4.0 DeepSeek code reads a different variable, DSV4_MOE_TOPK, so check which build you're on. Leave pruning off until you've seen the default output.

Honest limitations

  • The benchmarks are the maintainer's and the community's, not independent. The README labels rows "Verified" without saying who verified them or how.
  • The test conditions are thin. The README reports snapshot restore times of 18.9 milliseconds for the 30B model and 13.1 to 32.6 for the 284B model, and memory use as a 57.61 to 58.9 GiB range, without fully explaining the setup behind each number.
  • You run a prebuilt binary from a model repo. The direct-run path downloads an executable server from Hugging Face alongside the weights. Know where it came from before you chmod +x it, or build from source.
  • The license covers the code, not the model. Apache 2.0 applies to modchip's code. The DeepSeek weights and the converted files carry their own terms, so check those before you ship anything built on them.
  • The numbers come from main, not the tag. The release page shows main has moved past v2.4.0, so what you clone today may not match the tagged build.
  • 64 GB is the floor, not a comfortable fit. With roughly 58 to 59 GiB in use at full context, there is little room for anything else on the machine.
  • Lock-in is the default. The Fractal-64 files don't open in standard llama.cpp, Ollama or LM Studio.

What to do with this

If you own a 64 GB desktop and have wanted a frontier-size open model without buying a GPU, llama-modchip is a concrete, if self-reported, demonstration that the bottleneck for big mixture-of-experts models can move to system memory. That shift is worth understanding even if you never run this fork.

Run the 30B model this week. If it holds up on your hardware, decide whether a 56 GiB file that only one project can open is a trade you want. Keep your standard GGUF files (the usual llama.cpp model format) around either way.

Sources: llama-modchip README (main branch, read November 8, 2026), v2.4.0 release, Hacker News thread, llama.cpp.