FreeToken Runs a 753B Model on One Workstation GPU. The Real Trick Is That Your VRAM Split Moves at Runtime.

A Berkeley and MIT author list published an MoE serving engine that treats your desktop as an elastic pool instead of a small GPU. The parameter counts are the marketing. The memory model is the engineering.

A paper landed on arXiv on August 17 claiming a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on a single workstation GPU. The author list includes Kurt Keutzer, Song Han, Matei Zaharia and Ion Stoica. The code is Apache 2.0, it is on PyPI, and there is a one-click desktop app.

Those numbers are doing all the work in every writeup I have seen, and they are the least interesting thing in the paper.

FreeToken is a Mixture-of-Experts serving engine, and its actual argument is a refusal. It refuses to pick an offloading strategy. Every local inference stack before it decided, ahead of time, which layers live on the GPU and which live in host memory, and then you lived with that decision for the length of the run. FreeToken maps computation and model state onto whatever the machine exposes, continuously, and it can reallocate VRAM between the expert cache and the KV cache mid-session without restarting the engine or reloading weights.

That changes the question you should be asking. Not "does this model fit," which is a static question with a yes-or-no answer, but "what does my memory split look like at the context length I actually work at," which is a moving target you have to measure yourself.

What the engine is actually doing

The paper frames the design around two facts about local AI that datacenter serving stacks were never built for.

The first is that edge hardware is uneven in a way that varies per machine. A laptop with an 8GB GPU and 64GB of fast system RAM has a completely different bandwidth profile from a workstation with a 96GB card and slower host memory. The ratio between them, not the absolute size of either, decides which placement wins. FreeToken calls its scheduling policy q*, a bandwidth-adaptive split of work across CPU and GPU, and pairs it with full-layer double-buffered prefill streaming so weights arrive while compute is still busy. Expert residency is managed by a global LRU cache, which is the right structure for MoE specifically, because at any given token only a fraction of experts fire and the hot set drifts as the conversation moves.

The second fact is the one that matters if you point a coding agent at this. Agent workloads change their execution pattern constantly. A tool call comes back and gets spliced into the middle of the context. A thinking block gets edited. On a conventional serving stack, editing context at position N invalidates the KV cache from N onward, and you pay to recompute the tail on every turn. FreeToken ships what the README calls semantic anchor checkpoints, recurrent state and KV snapshots taken at meaningful boundaries so those agentic edits do not force full recomputation.

That is a serving engine designed by people who have watched an agent loop, not just a benchmark harness. The README lists Codex, Claude Code, OpenCode, OpenClaw and DeepSeek Harness as clients it works with, behind Anthropic-compatible and OpenAI-compatible APIs, which means you point your existing harness at a local endpoint and change nothing else.

Why the moving memory split is the headline

Here is the part I would put on a sticky note.

On a fixed-allocation engine, your model residency and your context length are decided once, at load. You pick a quantization, you pick a context size, the numbers either fit or they do not, and the failure mode is loud. On FreeToken, expert cache and KV memory share one pool and the boundary between them moves while you work.

That is strictly better ergonomics and strictly worse predictability. Better, because a short chat turn can hand VRAM to experts and run fast, and a 100k-token agent turn can claw it back for KV without you restarting anything. Worse, because the configuration that produced yesterday's throughput number is not necessarily the configuration running right now, and the thing that degrades when the split moves is not a crash. It is speed, and then, if the engine has to fall back harder than you expected, it is behavior.

So the number to write down is not "GLM-5.2 runs on my card." It is your tokens per second and your time to first token at your working context length, on your quantization, measured twice on different days. If those two measurements disagree, that is the elastic allocator doing its job, and you now know the size of the band you are actually operating in.

Put this into practice

The lowest-friction path is the desktop app. Download it for Windows or Linux from flashml.ai, which sets the engine up and gives you a GUI for pulling models, chatting and tuning. If you would rather stay in a terminal, one command:

uv pip install "freetoken[accel]"

Then start with a model you already know well, not the 753B headline. Run something in the Qwen3.6-35B-A3B class first, at a quantization you have used elsewhere, so you have a baseline in your head to compare against. Point Claude Code or Codex at the local endpoint using the Anthropic-compatible or OpenAI-compatible base URL and run a real task, not a chat.

Then do the measurement that actually tells you something. Take one agent task you run often, one that hits your normal context length and makes real tool calls, and time it end to end. Note tokens per second, time to first token, and whether every tool call closed cleanly. Run it again the next day from a cold start. The gap between those two runs is your operating band, and it is the number no model card will ever print for you.

If the desktop app exposes the expert cache and KV split, watch it during a long agent turn. Watching the boundary move while a task runs is the fastest way to understand what this engine is doing that others are not.

What to know before you commit

The hardware story is narrower than the paper's framing suggests. Native support is NVIDIA RTX 30, 40 and 50 series. No Apple Silicon, no AMD, and the desktop app is Windows and Linux only. If you are on a Mac, this release is not for you regardless of how much unified memory you have.

The headline sizes assume hardware most people do not own. A 753B model on a single workstation GPU means a workstation GPU. The gaming-desktop figure is where most readers live, and there is a small discrepancy worth naming: the repo's own tagline advertises running "290B+ frontier MoE models locally on your gaming PC," while the paper says 284B on a gaming desktop. It is rounding in the friendly direction, and it is the kind of rounding you should not size a purchase against. Read the paper's numbers, not the banner.

The project is young in a way the star count hides. When I checked the repo it showed 4.3k stars and 391 forks against 37 commits, 70 open issues, 54 open pull requests, and no tagged releases. PyPI is at 0.1.2. The paper is a v1 preprint with no venue attached. Every one of those is normal for a two-week-old research release and every one of them is a reason to keep this beside your current setup rather than replacing it.

One more, and it is specific. The supported quantization list includes NVFP4. Independent precision testing published last week found NVFP4 finishing last of five quantization schemes on top-1 token agreement in long agentic contexts, with tool calls failing to close. That is a finding about a quantization format, not about FreeToken, and FreeToken also supports MXFP4, FP8 and BF16. But an engine whose whole value proposition is aggressive adaptive placement across uneven hardware is exactly where you want to be conservative about format choice. Start at the highest precision your card will hold and walk down deliberately, verifying tool calls at each step.

FreeToken also credits its lineage openly, naming mini-sglang as direct inspiration and SGLang, vLLM, FlashInfer, LightLLM and llama.cpp as sources of design and reused code. That is a good sign about the maintainers, and a fair signal about maturity: this is a sharp new arrangement of well-understood parts, not a stack with years of production scars.

The reason to try it anyway

For two years the answer to "can I run frontier open weights locally" has been a hardware question with a boring answer. FreeToken makes it a scheduling question, and scheduling questions get better every release.

Install it beside what you already run. Give it one real agent task and two measurements on two different days. If the band between those measurements is tight enough for your work, you have just moved a class of models onto hardware you already own, and the thing you should keep watching is not the parameter count. It is where the memory boundary sits when the task gets long.

Sources: FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution, arXiv:2608.16157; FlashML-org/FreeToken on GitHub; flashml.ai.