Independent AI intelligence Two editions daily · ET
Fervor AI
Menu

Analysis · September 14, 2026 · concept

AutoRoundArchslothllama.cppQwen3local-aifine-tuningagent-infrastructure

Same Bytes, Different Model: What AutoRound's Calibration Corpus Does to a GGUF Quantization

Two Qwen3-4B Q4_K_M files, 2,497,280,800 bytes each, identical tensor layout, 54 percent apart on the only measurement that matters

Two files sit on Hugging Face with the same name, the same size to the byte, and the same tensor map: 216 tensors in Q4_K, 37 in Q6_K, 145 in F32, zero differences. Load either one into stock llama.cpp and it runs at the same speed. By every number a model card usually shows you, they are the same model. Measured as KL divergence from the bf16 original, one of them is 54 percent closer on Korean text and 33 percent closer on English. The only thing that differs is which integer each weight was rounded to, and the thing that decided the rounding was a text file.

That is the finding in a post the Archsloth team published on Hugging Face on September 14, and it is unusual for two reasons. It is a quantization shop publishing the two flags it had set wrong for months, and it is a controlled experiment where the thing being tested is not bit width, not layout, not a fork of the runtime, but the calibration corpus alone.

Why this matters more than another leaderboard

If you run local models, you pick GGUF files by name. Q4_K_M means a specific llama.cpp layout, and the assumption baked into every download is that two Q4_K_M files of the same base model are interchangeable up to noise. The Archsloth pair breaks that assumption in a way you cannot see from the file. Same bytes, different model.

My position: the file name and size are no longer enough to know what you are deploying, and the tool most people reach for to check (perplexity) does not show the difference. On the same two files, perplexity moved 2.8 percent. KL divergence moved 54.4 percent. If you rank quantizations by perplexity, the post argues, you are mostly ranking noise. I think that is right, and the mechanism below is why.

The two flags

Intel's AutoRound is a post-training quantization toolkit built around a rounding search: instead of rounding each weight to the nearest representable value, it runs a sign-gradient descent over a calibration set to decide, weight by weight, whether rounding up or down reconstructs the layer's activations better. The Archsloth team had been running it with --scheme W4A16, the 4-bit-weight, 16-bit-activation scheme built for GPU tensor cores, and then exporting with --format gguf:q4_k_m.

Those are different quantizers. W4A16 and Q4_K_M group weights differently, derive scales differently, and place the zero point differently. So the search was minimizing error for a quantizer that was never the one used at export, and the export then re-derived everything from scratch. Most of the search was thrown away. Nothing failed. The loss curve went down, the GGUF was valid, and the resulting file was simply no better than plain llama-quantize.

The fix is one flag, --scheme GGUF:Q4_K_M, which AutoRound's own README lists as the recommended GGUF scheme. The second change is --enable_alg_ext, which turns on the SignRoundV2 search (arXiv 2512.04746). It is off by default, AutoRound labels it experimental, and it costs about 1.7 times the tuning time: eleven minutes instead of six on a 4B, on one H100. Those two lines are the entire difference between the two columns.

Here is the corrected build, verbatim from the model card:

auto-round --model Qwen/Qwen3-4B \
  --scheme GGUF:Q4_K_M --enable_alg_ext \
  --iters 200 --nsamples 128 --seqlen 2048 \
  --dataset cal_archsloth.jsonl \
  --format gguf:q4_k_m --output_dir out

The lever nobody prices in

The part of the post I keep coming back to is the calibration corpus. There is a well-known result that importance-matrix (imatrix) calibration language barely matters, and the post agrees it is correct. It just does not transfer to AutoRound, and the reason is mechanical.

An imatrix reweights which weights matter inside llama.cpp's own scale search. The search still runs; the matrix tilts it, and its influence is bounded by that search. AutoRound's rounding search replaces that search. For each weight it decides up or down by minimizing reconstruction error on the calibration activations, and there is no second procedure downstream to wash the choice out. So the text you calibrate on does not nudge the result. It chooses the rounding direction.

The controlled pair makes the point. Same model (Qwen3.5-9B), same code, same flags, same bit widths, same byte count, only the corpus changed. English-only calibration: Korean KL 0.027336. Korean plus English, interleaved at the sample level: 0.019255. That is 29.6 percent of Korean quality from the text alone. On the 27B at Q4 the same comparison gives 31.9 percent.

For scale, the knob the field spends effort on, per-layer bit allocation, bought between nothing and 10 percent worse on top of this recipe. The text file was worth thirty percent; the bit budget was worth nothing.

Two smaller findings from the CALIBRATION.md that ships in the repo are worth knowing before you build your own. Interleaving mattered more than the ratio: moving to strict alternation at the sample level, with the same overall character mix, improved English by 11.4 percent, because the rounding search consumes samples in order and a run of same-language samples biases it before it reaches the others. And the tool's default corpus, NeelNanda/pile-10k, was not a neutral choice: against the mixed corpus it was 76 percent worse on Korean and 9 percent worse on English. Reaching for the default cost English, not just Korean.

Why perplexity hides it

Perplexity asks whether the model is still confident. KL divergence asks whether it is confident about the same things. For quantization, the second question is the one that matters, because the goal of a quantized file is to reproduce the original's output distribution, not to be a confident model in its own right.

On the Korean axis, against the same bf16 teacher, the numbers are: bf16 perplexity 8.605807; Archsloth Q4_K_M 8.694627 with KL 0.024297; Unsloth Q4_K_M 8.948656 with KL 0.053290. Perplexity calls the gap 2.8 percent. KL calls it 54.4 percent. Same two files. llama-perplexity reports both statistics in one run, so if you have ever measured perplexity on a quant, you have had the better number sitting in the log the whole time.

Put this into practice

The lowest-friction move is to stop trusting file names and start measuring the file you actually deploy, on text that looks like your workload. The recipe is four commands and needs the bf16 reference resident on a GPU (about 8 GB for a 4B).

# 1. the comparison file, straight from its own repository
hf download unsloth/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir rival

# 2. the reference the distance is measured from
python llama.cpp/convert_hf_to_gguf.py Qwen/Qwen3-4B --outtype bf16 --outfile ref-BF16.gguf

# 3. teacher logits, once per evaluation text
llama-perplexity -m ref-BF16.gguf -f eval/eval_ko.txt \
  --kl-divergence-base base_ko.dat -ngl 99 -c 512 --chunks 60

# 4. every candidate against the same teacher, same text, same chunks
llama-perplexity -m <candidate>.gguf -f eval/eval_ko.txt \
  --kl-divergence --kl-divergence-base base_ko.dat -ngl 99 -c 512 --chunks 60

Swap eval_ko.txt for a few thousand tokens of your own prompts, tool-call transcripts, or code. The Archsloth evaluation texts are in the repo's eval/ directory if you want to reproduce their numbers first.

Second, add the monotonicity check to whatever you already do. Build or download the Q4, Q5, Q6, and Q8 of the same model and confirm KL falls as bits rise. The post's authors found their own Q5_K_M scoring worse than their Q4_K_M, traced it to a 5 to 6 percent deviation between optimized weights and packed tensors in their toolchain, and refused to ship it. A file that gets worse when you give it more bits is a defect, not a result, and the check costs nothing.

Third, if you quantize your own models with AutoRound, evaluate every extra lever with the rounding search both off and on. Healing, mixed precision, and per-layer bit allocation all helped against plain round-to-nearest (6 to 37 percent better) and all hurt once --enable_alg_ext was on (9 to 57 percent worse). A lever measured only against a weak baseline will look like a discovery and then cost you in the shipping build.

Fourth, if your users write in a language other than English, put that language in the calibration corpus, interleaved, and keep evaluation axes for languages you did not calibrate on. Seven of Archsloth's ten evaluation axes are not in the corpus at all. They are there so that a build which buys one language with another has somewhere to show it.

Honest limitations

Every number here is self-run by the people selling the result. They ship the logs, the corpus, and the evaluation texts, and they name the exact rival file so you can fetch the bytes they measured, which is more than most quant releases do. It is still one team's measurement until someone replicates it.

The ten-axis sweep exists only for the 4B. The 9B and 27B were measured on two axes, Korean and English. The 27B Q4_K_M lost English by 15.8 percent against Unsloth's UD-Q4_K_M and was not published, which is honest and also means the recipe does not win everywhere.

KL over 60 chunks of 512 tokens measures distribution drift on short contexts. The post lists long-context retrieval and agentic tool use across bit widths as unmeasured. Their own 13-item format-compliance check scored Archsloth 12 of 13 and the same-size comparison file 11 of 13, and the authors say thirteen items cannot rank four models. If your workload is tool calling, none of the headline numbers speak to it yet.

The gain narrows as bits go up: 54 percent at Q4, 41 percent at Q6, 11 percent at Q8 on Korean. At Q8 you are arguing about a rounding error.

And the whole result depends on --enable_alg_ext, a flag AutoRound's README marks experimental and documents mainly for MXFP4 and W2A16. It worked here; the README does not promise it will.

The file is the claim

The interesting thing about this post is not that one shop beat another on Korean. It is that the field has been treating "Q4_K_M of Qwen3-4B" as a stable object when it is the output of a search whose most important input, the text it read, nobody states. Archsloth stated theirs and shipped it as cal_archsloth.jsonl. You can read it, diff it, or replace it.

The move available to you is smaller than building a quantization pipeline. Take the file you already run, take the bf16 it came from, and spend an hour finding out how far apart they are on your own text. Whatever number comes back, you will know something about your deployment that the file name never told you.

Sources: Archsloth post on Hugging Face (September 14, 2026), Archsloth/Qwen3-4B-GGUF model card and CALIBRATION.md, intel/auto-round README, SignRoundV2, arXiv 2512.04746.

Medium metadata

  • Title: Same Bytes, Different Model: What AutoRound's Calibration Corpus Does to a GGUF Quantization
  • Subtitle: Two Qwen3-4B Q4_K_M files, identical size and tensor layout, 54 percent apart on KL divergence
  • Tags: Local LLM, Quantization, llama.cpp, Open Source AI, Machine Learning
  • Canonical: fervorai.dev article URL once published
  • Hero image: two identical-looking file icons on a bench scale, one side lower, Ralph Steadman style, no text