Independent AI intelligence Two editions daily · ET
Fervor AI
Menu

Analysis · October 29, 2026 · repo

Edge0Qwen3.6-35B-A3Blocal-aifrontier-models

Edge0 Runs a 35B AI Model With a 3 GB Memory Peak by Streaming It From Your SSD

An open-source engine streams model weights from disk on demand. Here's how it works, what it costs you in speed and quality, and how to try it on an Apple Silicon Mac.

Local AI has lived by one rough rule: the model has to fit in memory. A 35-billion-parameter model squeezed to 4-bit precision still weighs about 19.5 GB, which rules out most laptops before you type a word.

That rule is starting to bend. Edge0, the top repo on Trendshift's GitHub trending board this morning, runs a 35B-class model on a Mac with a peak of about 2.9 GB of model memory. It does it by leaving most of the model on the SSD and reading pieces in only when they're needed. (That figure is the model's own memory peak, not a machine requirement: the project's tests ran on 16 GB and 24 GB Macs.)

The result is a trade worth understanding before you install anything. RAM stops being the wall. Your disk speed and a small prediction model take its place.

What Edge0 is

Edge0 is an open-source inference engine (the software that runs a trained model and produces its answers) built for a specific kind of model called mixture-of-experts, or MoE. The engine is Apache-2.0 licensed (the model card lists Apache 2.0 for the 35B weights too), had about 8.7k GitHub stars as of October 29, and tagged a new Android build, v0.1.1, the same day. The authors describe the method in a technical report on arXiv, submitted September 16.

It ships two models, both labeled preview:

  • edge0-35b, built on Qwen3.6-35B-A3B, about 23 GB on disk.
  • edge0-8b, built on Ling 3.0, about 4.2 GB on disk.

The Python framework runs on macOS with Apple Silicon (M1 through M4) using MLX, Apple's machine-learning library. There are also native engines for iOS, macOS, Android and Windows in the repo, which the model card calls "still maturing."

Why mixture-of-experts makes this possible

A regular model uses all of its weights for every word it writes. A mixture-of-experts model doesn't.

Think of a kitchen with 256 specialist cooks. For any given dish, a head chef (the router) picks a handful of them, and the rest stand around. The "A3B" in Qwen3.6-35B-A3B tells you how lopsided this is: the model has about 35 billion parameters in total, but roughly 3 billion are active for any one token.

So most of the model is idle at any moment. The question Edge0 answers is: why keep idle cooks in the kitchen at all?

How Edge0 works, in three moves

1. Experts live on the SSD. Edge0 keeps the expert weights on disk and reads them in on demand, so peak memory depends on the experts in use rather than the whole model. The README describes the Python path as memory-mapping the weights (letting the operating system read file pages only when they're touched). It's the difference between carrying a whole library around and fetching the three books you need.

2. A prerouter guesses what's next. Fetching from disk is slow compared with memory. If the engine waited until the router chose experts and only then started reading, every token would stall. Edge0 adds a small trained head, the prerouter, that predicts the next layer's routing one step ahead, so the disk read overlaps with the math already happening. Think of a waiter who starts plating your next course while you're still eating this one. The README claims this adds "up to +59%" decode speed, with bigger gains on slower storage and bigger models.

3. A LoRA patch wins back quality. Squeezing weights to 4-bit loses accuracy. Edge0 trains a LoRA (a small add-on set of weights that adjusts a frozen model) by having the 4-bit model learn from the full-precision original. The adapters stay separate from the base, so one read-only base file can serve several adapter sets.

The README is clear that the LoRA and prerouter adapters are both required. Leave one out and edge0 fails with a message, unless you pass --no-prerouter or --no-lora to run the plain base model.

The numbers that change your decision

Every figure here comes from the project's own measurements on one machine, a Mac mini M4 Pro with 24 GB, using a 3.3k-token prompt. Read them as a starting point, not a promise.

Memory. About 2.9 GiB peak for edge0-35b and about 1.0 GiB for edge0-8b, at short contexts. That's MLX's allocator peak (the most memory MLX itself reserved at once), not total process memory, and the README says the same build runs on a 16 GB M2. Long conversations grow the KV cache (the model's running memory of the conversation so far), so the card warns you need shorter contexts to stay near 3 GiB.

Speed. edge0-35b decodes at 14.9 to 17.7 tokens per second, and edge0-8b at 23.9 to 25.3. That's comfortable reading speed for both.

The wait before the first word. Prefill (reading your prompt before answering) is where the disk shows up. For edge0-35b it runs at 113 tokens per second cold and 140 warm. On a 3.3k-token prompt, that's roughly half a minute before the answer starts on a cold run. The 8B tier prefills at 500 cold and 1,428 warm, so it feels far snappier.

"Cold" and "warm" are worth a sentence. On a cold run the weights come off the SSD. On a warm run, macOS has kept recently read file pages in spare memory (the page cache), so the second prompt is faster. You're borrowing free RAM after all, just in a way you can give back.

Quality. The authors ran the same OpenCompass benchmarks (an open-source suite of standard tests) on Edge0 and on the full-precision base models. edge0-35b averages 79.2 against 83.2 for its base, about four points lower. The biggest drop is on AIME 2026 math, 86.6 against 92.7. The README also says MMLU-Pro came out "even above the base," which is true for the 8B tier (70.1 against 65.8) but not for the 35B (81.0 against 84.6).

The shift worth noticing

Here's the idea to carry away: when experts stream from disk, RAM stops being the limit, and the SSD and the guesser become the limits.

That changes how you shop and plan. A Mac with a fast internal SSD and modest RAM becomes a reasonable home for a 35B-class model. A slow external drive becomes a real problem, and the model card recommends fast storage such as NVMe (the fast SSD interface in modern Macs and PCs) or internal flash. And because the prerouter's guesses decide how often you stall, the quality of that small predictor matters as much as raw disk speed.

It also explains why the first prompt of the day feels sluggish and the second doesn't. That isn't a bug. It's the design showing.

Put this into practice

You need an Apple Silicon Mac, Python 3.10 or newer (3.12 recommended), and about 23 GB free for the 35B model or 4.2 GB for the 8B. These commands come straight from the README.

1. Install. From a clone of the repo:

cd python
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'

The README pins specific MLX versions (mlx==0.30.6, mlx-metal==0.30.6, mlx-lm==0.31.0). Keep them. Newer or older MLX builds are the first thing to suspect if output looks wrong.

2. Download a model. Start with the 8B if you want a quick first run:

.venv/bin/python scripts/fetch_models.py --tier edge0-8b --target-dir models
.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models

Then tell edge0 where the models live. The short tier names below resolve through these two variables; without them, pass the folder path instead (edge0 demo models/edge0-35b):

export EDGE0_35B_MODEL=$PWD/models/edge0-35b
export EDGE0_8B_MODEL=$PWD/models/edge0-8b

3. Try it. The demo and one-shot chat:

edge0 demo edge0-35b
edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."

4. Serve it to your other tools. edge0 serve edge0-35b starts an OpenAI-compatible endpoint at /v1/chat/completions, so anything that can talk to the OpenAI API format can point at it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"edge0-35b","messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'

5. Measure on your own machine. The repo includes the benchmark it used, so you can check whether your SSD keeps up:

python examples/bench.py edge0-35b

What to expect: a slow first prompt while weights come off disk, a faster second one, and memory use far below what a 35B model normally needs. If output comes back garbled and mixed-language on an A18 or A18 Pro chip, the README traces it to the wrong MLX version and recommends reinstalling the pinned mlx==0.30.6 and mlx-metal==0.30.6.

Honest limitations

It's Mac-only for now, outside the native apps. The Python framework supports macOS on Apple Silicon and nothing else. The CUDA backend is a reserved folder, planned for the fourth quarter of 2026.

It's a preview. Both models carry the preview label, and the card says coverage and quality are still being extended.

It's weak at agent work. The model card says agent capabilities are weak in this release, naming tool use, multi-step planning and long-horizon autonomy. If you wanted a local brain for a coding agent, this isn't it yet. It's a solid local chat and reasoning model.

The numbers are the project's own. Speed comes from one machine. Quality comes from the authors' benchmark runs. The arXiv abstract mentions 20 tokens per second on a 24 GB machine, while the README's table shows 14.9 to 17.7, so trust your own bench.py run over either.

You give up some quality. About four points on average, more on hard math. For many tasks that's fine. For some it isn't.

Where this leaves you

For years, the question for local AI was "how much RAM do I have?" Edge0 swaps it for two better questions: how fast is my disk, and how often does the engine guess right?

If you have an Apple Silicon Mac with a fast internal SSD, the 8B tier takes a few minutes to try and tells you most of what you need to know. Run the benchmark, watch the cold and warm numbers, and decide whether a slightly weaker 35B model with a 3 GB memory peak is worth more to you than a stronger one you can't run at all.

Sources: Edge0 README, main branch, read October 29, 2026; Edge0 v0.1.1 release feed; Edge0-35B-A3B-preview model card; arXiv 2609.18063, "The Other Half of the Memory Wall"; Trendshift daily board, read about 07:25 ET. Benchmarks are the project's own.