Top-1 Token Flips: How Your vLLM Backend and Quantization Choice Change What the Model Says
A lab test on Qwen3.6-27B held the GPU, the weights, the prompt and the sampler constant, changed one vLLM config flag, and watched the model target the wrong router interface. Here is how to run the same measurement on your own stack.
There is a complaint that shows up on every local AI forum, phrased almost identically each time. Someone reads that a model scores well, downloads it, runs it, and finds it noticeably worse than advertised. The standard explanations are quantization damage and wrong sampler settings, and both are real causes. A series of experiments posted to the Level1Techs forums by user thr3e in mid-August found something narrower and considerably stranger.
Change one line of inference configuration. Same GPU, same driver, same vLLM build, same weights, same prompt, same everything. The model now targets GigabitEthernet0/1/4 when the correct interface was GigabitEthernet0/0/1, then runs show run twice trying to dig itself out.
That is not quantization. That is not temperature. That is the choice of attention kernel.
The claim under the numbers
The write-up (part one here) captures full-vocabulary logits from Qwen3.6-27B on an RTX PRO 6000 Blackwell, tensor parallelism 1, BF16 weights, BF16 KV cache, eager execution, CUDA graphs and prefix caching disabled. The prompt is roughly 100k tokens of a real network-automation workstream with multiple tool calls, chosen because it appears in no public benchmark and nobody could have tuned a quant against it.
The metric is top-1 agreement: given the same forced token history, does the highest-scoring next token match the reference? A top-1 flip means a given configuration would have picked a different word.
Swap only the full-attention backend between FlashAttention 2, Flash Inference and Triton. Nothing else. For the first several thousand tokens, all three agree completely. Then they start disagreeing, in clusters, tied to prompt content rather than growing smoothly with length.
Here is the part that closes the door on hand-waving. Repeat runs of the same backend produced logits that were bit-for-bit identical at every hidden state. There is no sampling noise in this result and no randomness to blame. The divergence comes out of the matrix multiply and accumulate operations inside the kernels, and nothing else.
The weight-precision bakeoff is where it gets expensive. Five checkpoints of the same model, KV cache forced to BF16 across all of them:
- BF16 reference (
Qwen/Qwen3.6-27B) - Official FP8 (
Qwen/Qwen3.6-27B-FP8) - A community INT8 W8A16 quant (
TheHouseOfTheDude/Qwen3.6-27B-INT8) - NVIDIA's NVFP4 release (
nvidia/Qwen3.6-27B-NVFP4) - AWQ W4A16 (
cyankiwi/Qwen3.6-27B-AWQ-BF16-INT4)
The community INT8 beat the first-party FP8. NVIDIA's NVFP4 finished last, hitting roughly 50 percent top-1 disagreement by 88k of context. Both NVFP4 and AWQ W4A16 failed to close their tool calls and botched Cisco syntax, running show run where the correct command was show arp. FP8 and INT8 completed their calls correctly.
One caveat the author flags himself and that matters if you are shopping for hardware: vLLM classified the GPU path as lacking native FP4 support in that nightly build and fell back to weight-only FP4 compression through Marlin. The NVFP4 result measures that path, not native FP4 arithmetic.
The story repeats one layer down. In part two, at tensor parallelism 1 a tool call succeeds, at TP2 it fails, at TP4 it succeeds again, and the author traces that pattern to NCCL. In part three, four abliterated Qwen3.8 fine-tunes get the same treatment. Heretic-ARA and Huihui land near stock (0.717 to 1.406 percent top-1 flips depending on prompt). AEON Ultimate hits 5.831 percent on the longer prompt, and produces the failure worth remembering: with PostgreSQL port 5432 sitting in context, stock picked the final 2 with probability 0.9991, AEON picked ql with probability 0.9158, and the continuation rendered the port as 543ql before failing to close the tool envelope. The author is careful that AEON bundles SSM repair, an Abliterix search and an MTP graft, so none of that damage can be pinned on abliteration alone.
Why this is a serving problem, not a model problem
The mental model most of us carry is that a model has a quality, the model card reports it, and quantization subtracts some of it. That model is close enough for chat and wrong for agents.
An agent's output contains literals. Interface names, port numbers, hostnames, API argument names, SQL. A prose answer degrades gracefully because a synonym is still a synonym. A literal does not degrade. 5432 and 543ql are not similar values, they are a working command and a broken one, and the failure surfaces downstream inside a tool call where nobody is reading the logits.
The five checkpoints above are five confident model cards for the same weights. Three of them close a tool call. That gap is not visible in any published benchmark score, because the benchmark measured the reference implementation on the lab's hardware and software, and you are not running either.
Something else worth sitting with: when stock Qwen was confident about the next token, the well-behaved derivatives did not overturn it. The flips clustered where the model was already uncertain. Only the most aggressive edits overruled high-confidence tokens, and those are exactly the cases that produced operational damage.
Putting this into practice
You do not need an RTX PRO 6000 and 44 terabytes of NVMe to act on this. Start with the cheapest steps.
Read the model card's sampler settings and actually use them. Temperature, top-p and the chat template vary by model, and the card usually states them. Setting temperature too low is why a Qwen model sits looping inside its thinking block. This is free and most people skip it.
Pin your inference stack and write down what you pinned. Attention backend, tensor parallelism, KV cache dtype, CUDA graph settings, vLLM build. If you cannot reproduce a good day, you cannot debug a bad one. The nightly container in this study carried 734 packages, each with its own quirks, and the path your setup takes through that pile is specific to you.
Change one variable at a time when you evaluate. The entire value of this study comes from that discipline. Swapping the quant and the backend together tells you nothing about either.
Build a prompt from your own work, not from a benchmark. Capture a real session with real tool calls, scrub anything sensitive, and keep it. The author uses a model router to capture live workstreams for exactly this purpose. A prompt that has never appeared publicly cannot have been trained on or calibrated against, which makes it the only honest test you own.
Grade on tool calls, not on vibes. Did the command parse. Did the tool envelope close. Did the literal survive. Those are boolean and cheap to check, and they are where the damage lands.
Treat KLD claims on quant cards as unreadable without methodology. A low divergence number means nothing unless the author discloses the reference checkpoint, the runtime, the evaluation text, the calibration data, the context lengths, the sampled positions, the KL direction and the aggregation. Most cards disclose none of it.
If you want a starting bias from this data: on the tested workload, W8A16 held up better than W8A8, and both four-bit options broke tool calls. That is one model, one prompt, one GPU family. Do not generalize it into a rule.
What this study does not show
The author is unusually honest about his own limits, and the honest reading requires carrying those forward.
Part one sampled about 3 percent of positions, one probe every 32 tokens. He says plainly that this is not a correct methodology for characterizing divergence and was meant only as a high-level glimpse. Part two moved to 100 percent capture in targeted ranges precisely because of that.
BF16 is a numerical-fidelity reference, not an oracle. A quantized checkpoint can diverge from BF16 and produce a better answer. Nothing here labels the reference correct.
A top-1 flip is a counterfactual root, not a different final answer. Under teacher forcing, most flips get absorbed. That is why the branching analysis in part two exists, and why "invalid branch futures" counts structurally broken output among specifically chosen divergence roots rather than a general tool-call failure rate.
The flip rate is highly prompt-dependent. Some tasks sit under 1 percent across every configuration. Others spike. There is no universal context length where a model falls apart, and anyone quoting one from this data is overreading it.
And this is one model family on one GPU generation, run by one person with a lab and a grudge. H200 and B200 results were still landing when this was written. The tooling and dataset are not yet packaged for other people to run, which is the single thing that would turn a compelling forum series into something you can cite.
The part you can use tomorrow
The useful shift here is small and slightly uncomfortable: the quality of a local model is a property of your serving stack, and the model card is a claim about somebody else's.
You already accept this everywhere else in your infrastructure. Nobody reads a database vendor's benchmark and skips load testing. Local inference got a pass on that because the weights felt like the whole product, and they are not. The weights are one input to a stack of kernels, cache formats, collective operations and drivers that decide, deterministically and repeatably, which token comes next.
Capture one real session. Run it through two configurations. Count the tool calls that survive. You will either confirm your setup is fine, which is worth knowing, or you will find out why the model has felt slightly off for a month.
Sources: Why your local LLM feels dumber than it is, Level1Techs forums (parts 1 through 3, August 16 to 20, 2026); Hacker News discussion; model cards for Qwen/Qwen3.6-27B, Qwen3.6-27B-FP8 and nvidia/Qwen3.6-27B-NVFP4.