Quantization Damage Is Nonlinear, and Qwen3.8 27B Shows Exactly Where the Cliff Is
Four-bit matched the full model on an agentic coding benchmark. One-bit scored below random guessing. The interesting part is what happens between them.
A one-bit build of Qwen3.8 27B answers graduate-level science questions worse than a coin. Not worse than the full model, worse than chance, which means something stranger than degradation is happening: the model is not producing bad answers, it is producing no answer at all, reasoning until the token budget empties and returning nothing. Give it more thinking room and the score drops further. That is the finding at the bottom of Piotr Migdał's quantization benchmark, published on August 26 and pushed to the Hacker News front page on September 8, and it is a much more useful number than the one everybody quotes from these posts.
The number everybody quotes is the top of the chart. A 17 GB four-bit build matches the 55 GB BF16 original on Terminal-Bench 2.1, a real agentic coding benchmark with 89 tasks, and fits a 24 GB card with room for roughly 64k of context. That is a great headline. It is also the least actionable thing in the piece, because it tells you where the curve is flat and gives you no idea where it stops being flat.
Here is the position I will defend for the rest of this article: the standard advice, run the biggest model that fits in your VRAM, is wrong on the wrong axis. Size is not the variable. The shape of the damage curve is, and that shape is nonlinear in a way that makes interpolation actively dangerous. You cannot reason from "four-bit is fine and one-bit is broken" to any conclusion about two-bit. You have to measure, and the measurement has to be your task, not a proxy for it.
Why the usual proxies lie to you
The convenient way to measure quantization damage is to compare token predictions against the full-precision model: KL-divergence, or the share of positions where the top-1 token still matches. It is cheap, it produces a smooth curve, and Unsloth publishes exactly this kind of figure, advertising its 6.2 GB one-bit build as retaining around 72% top-1 accuracy while being 89% smaller.
Read that as a practitioner and it sounds like a modest tax. Twenty-eight percent of tokens differ. Most of them are probably paraphrase. And on GPQA Diamond, the same model scores at random guessing.
Both facts are true at once, and the gap between them is the whole problem. Token agreement is a per-position average over a distribution where almost every position is easy. The positions that decide whether a task succeeds are not average positions. They are the branch where a logical step goes one way or the other, the closing brace, the tool name, the moment the model decides whether it has finished. A quantization that gets 72% of tokens right can still get zero percent of the load-bearing ones right, because the load-bearing ones are exactly the low-margin decisions that quantization noise flips.
Migdał's response to this is the right one and it costs money: skip the proxy, run the actual benchmarks. He ran GPQA Diamond, IFBench and Terminal-Bench 2.1 across the ladder with llama.cpp, and posted the bill, roughly $3,000 of rented GPU time on Modal, itemized per benchmark. Terminal-Bench alone came to $2,308.
That price tag is not a footnote. It is the reason this measurement almost never gets done, and the reason most quantization advice you will read is somebody's vibe from a Reddit thread. Three thousand dollars is a lot to spend proving that a 17 GB file behaves like a 55 GB one.
The shape of the curve
The curve has three regimes, and knowing which one you are in matters more than knowing your bit depth.
The flat regime runs from full precision down through four-bit. On GPQA Diamond and IFBench there is no measurable movement outside noise. On Terminal-Bench 2.1 the four-bit build reproduces the full model's reported score. Migdał also replicated Qwen's own published numbers for the BF16 model on the first attempt, which is worth pausing on, because benchmark replication usually fails on hidden settings and the fact that it did not gives the rest of his numbers a lot more weight.
The soft-decline regime is two-bit. Scores dip. On Terminal-Bench the two-bit build falls noticeably but lands around the level of Opus 4.7 or Gemini 3.1 Pro, which is to say far from frontier and also far from useless. It costs something in process, too: on the same solved tasks it uses roughly a quarter more output tokens than the full model for the same number of turns. It thinks harder to get to the same place.
The collapse regime is one-bit, and it does not look like a steeper version of two-bit. It looks like a different failure mode. The score sits at chance, and the interaction with reasoning effort inverts: at the model's default xhigh setting the one-bit build scores below its own low setting, because longer reasoning traces mean more chances to wander until the budget runs out and the answer comes back empty. That inversion is the tell. A model that is merely less accurate degrades smoothly with more thinking time. A model that has lost coherent control of its own reasoning gets worse.
Which brings up the setting nobody budgets for. Qwen3.8 27B ships with reasoning effort defaulting to xhigh, and Simon Willison's write-up from August 16 documents what that means in practice: 22,276 reasoning tokens and 21 minutes to draw one pelican on a bicycle, and a request for "an svg of a circle" that produced a beautiful animated geometric study nobody asked for. His recommendation was blunt, start on low or with reasoning off. Migdał's data adds the other half: effort interacts with quantization, and the direction of that interaction flips at the bottom of the ladder. You are not tuning two independent knobs.
Put this into practice
Here is the procedure I would actually run, in order, and it does not require $3,000.
Budget the KV cache first, not last. Migdał used F16 KV cache throughout, which costs roughly 2.3 GB per 32k tokens on this model. On a 24 GB card holding a 17 GB model, that is about 64k of context and no more. If your workload needs 128k, you are choosing a smaller quant whether you meant to or not. Do this arithmetic before you pick a file, because it silently decides the answer.
Download the file you are going to run, then benchmark that file. Unsloth replaced the v2 GGUFs on August 19, mid-benchmark, so the exact artifacts behind most of Migdał's numbers are no longer downloadable. The Hugging Face listing today shows UD-Q4_K_M at 16.5 GB, UD-Q2_K_XL at 9.83 GB, UD-IQ1_S at 6.19 GB and BF16 at 54.7 GB, which are close to but not identical to the sizes in his post. Quant files are not stable artifacts. Treat any published quantization benchmark as evidence about a method, not about a specific download.
Run three arms, not five. Full precision if you can borrow the hardware for an hour, your candidate quant, and the next one down. You are not mapping the curve, you are locating which regime your candidate sits in. Two adjacent points that agree means flat. Two that diverge means you are already on the slope and should back up.
Make the benchmark your actual task. Terminal-Bench 2.1 was the discriminating test in this data; the one-shot benchmarks flattened out earlier. If your workload is agentic, a knowledge quiz will tell you nothing, and if your workload is extraction, an agentic benchmark will overstate the damage. Twenty representative tasks from your own backlog with a pass or fail on each will beat any public leaderboard for this purpose.
Pin reasoning effort as part of the configuration, not as a runtime preference. Test each quant at the effort level you will ship, and test at least one lower level. If the lower level scores higher, that is your signal that this quant has lost the plot, and no amount of prompt work will fix it.
Check that pathological empties are counted as failures. The one-bit collapse is partly invisible to any harness that treats a blank response as a skip. If your eval discards empty completions without flagging them, you will see a small accuracy dip where the real result is a model that stops answering.
Where this evidence stops
One model, one quantizer, one runtime. Everything above comes from Unsloth GGUFs of a single 27B dense model run under a llama.cpp build from August 16, and dense models behave differently from mixture-of-experts models under quantization, because MoE routing concentrates damage in the router. Do not carry the cliff position across architectures.
The KV cache is untested here. Migdał holds it at F16 throughout and says explicitly that he has not measured KV quantization, while noting that people report it is more sensitive than weight quantization. Since KV cache is often what actually forces the tradeoff on a consumer card, that is a hole in the practical guidance, and it is his hole to fill, not mine to paper over.
The two-bit result is one point on one benchmark. A single noticeable drop on Terminal-Bench with 89 tasks is not a fine-grained claim about where the slope begins, and Migdał says he skipped the eight-bit Terminal-Bench arm by accident, then chose to interpolate rather than re-run it, on the grounds that the run would be costly and unnecessary.
And the one-bit finding is a finding about one-bit builds of this model in August 2026. Unsloth's v3 method is newer than the v2 files used for most of these tests, the vendor claims it beats other providers at equal size, and nobody has independently re-run this ladder against v3. That is worth saying because the honest version of "one-bit collapses" is "one-bit collapsed here, on these files, and the method has since changed."
What to do with this
The reason this benchmark travelled is not that it found four-bit acceptable. Everyone assumed that. It travelled because it drew the line where the shortcut stops working, and put a dollar figure next to what drawing that line costs.
You will not spend $3,000. But you can spend an afternoon, and the afternoon is worth more than the chart, because the chart is about Qwen3.8 27B on a rented L40S and your afternoon is about your model on your card doing your work. Pick your candidate, pick the one below it, run twenty of your own tasks through both at your shipping reasoning effort, and count the empties.
If the two agree, take the smaller one and spend the freed memory on context. If they diverge, you have just learned that you were already standing on the slope, and that is the single most valuable thing a benchmark can tell you.
Sources: Piotr Migdał, "Benchmarking Qwen3.8 27B quantizations" · unsloth/Qwen3.8-27B-GGUF on Hugging Face · Simon Willison on Qwen 3.8 27B
Medium metadata
- Title: Quantization Damage Is Nonlinear, and Qwen3.8 27B Shows Exactly Where the Cliff Is
- Subtitle: Four-bit matched the full model on an agentic coding benchmark. One-bit scored below random guessing. The interesting part is what happens between them.
- Tags: local llm, quantization, llama cpp, machine learning, open source ai
- Canonical: import from the fervorai.dev URL