vLLM Constrained Decoding Fixes the Tool Calls Your Self-Hosted Agent Keeps Breaking

Self-hosted agent models fumble a surprising share of tool calls. Often the fix is a server setting you never turned on.

Agents fail in a way that rarely shows up in a demo. The model picks the right tool, means the right thing, and then writes the call slightly wrong: a misspelled argument, a missing field, a tool name that is close but not real. The tool rejects it, the error lands in the conversation, and every turn after that carries the mess.

On your own GPU, nothing stands between the model's guess and your parser unless you put it there.

That gap just got measured. Ubicloud, a cloud provider that serves open models, found that Kimi K2.6 running on vLLM 0.28 produced schema-correct tool calls only 92% of the time. Roughly 2 calls in every 25 failed. Turning on one feature, constrained decoding, took that to 100% on their benchmark.

The result is a shift in where you look when an agent misbehaves. Before you swap models or rewrite prompts, check whether your server is letting the model write invalid calls at all.

What a "broken tool call" actually is

A tool call is structured text. When an agent wants to search the web, the model writes something like a function name plus a small block of JSON with the arguments. Your agent framework parses that and runs the tool.

The tool's schema is the contract: which arguments exist, which are required, what type each one is. A call is "schema accurate" when it matches that contract.

Models write these calls one token at a time, by probability, like everything else they write. Usually the most likely tokens form a valid call. Sometimes they don't. Ubicloud's post gives the most relatable example: a model trained on a tool called search_web writing that name when your tool is actually web_search. Close enough for a human. Fatal for a parser.

The failure compounds in agents. One bad call puts an error message into the context window (the running transcript the model rereads every turn), and the model now reasons over noise. Ubicloud describes exactly this: failed calls pollute the context and degrade multi-turn work.

Constrained decoding, in plain terms

Think of the difference between a free-text field and a dropdown on a form. A free-text field accepts anything, so people misspell things. A dropdown only offers valid options, so they can't.

Constrained decoding turns the model's output into a dropdown. At every step, before the model picks its next token, the server masks out every token that would break the schema. The model still chooses, but only among tokens that keep the call valid. If the schema says the tool is web_search, the path that spells search_web simply isn't available.

In vLLM, the popular open-source serving engine for running open models on your own GPUs, this works through grammars, machine-readable rules for which text is allowed, built from your tool definitions (for tool calls, vLLM uses what it calls structural tags, built on the XGrammar library, per Ubicloud). The authors say the runtime overhead is negligible. The vLLM docs add one real cost: the first time a grammar is compiled there can be "several seconds of latency (or more)."

Why it was off for so many people

Here is the part that catches teams. vLLM added this feature upstream in v0.25, and Ubicloud switched from their own fork to it. Their accuracy then dropped from 92% to 91%. The post explains why: vLLM changed its default behavior for requests that do not explicitly turn constrained decoding on. So the feature existed, and it wasn't running.

The current vLLM tool-calling docs spell out the default. With --tool-strict-level auto (the default), vLLM follows each request. If your agent sends tool_choice="auto", which is what most agent frameworks send so the model can decide whether to call a tool, the grammar only switches on when at least one tool in the request is marked strict: true. And even then, the argument schema is pinned only for the tools that carry strict: true. A tool without it keeps free-form arguments.

Most tool definitions in the wild don't set strict. So most self-hosted agents run unconstrained without anyone deciding that.

Why not just flip the default? The Ubicloud post is candid about it. Constrained decoding is per-model work, and turning it on everywhere caused problems: some models' unconstrained output already drifts from the expected format, grammar translation has had bugs, and speculative decoding (a speed trick where a small model drafts tokens for the big one to accept) can wrongly reject tokens when both are on. The post cites users of gpt-oss-20b, Qwen 3.6 and GLM 5.1 who hit trouble when it was on by default.

So the off switch is a deliberate tradeoff by the vLLM maintainers, not a mistake. It just means the decision is now yours.

The two switches that matter

You can turn constraints on per tool, or set a floor for the whole server.

Per tool, in the request. Mark a tool strict: true. The docs say this works across Chat Completions, Responses and Anthropic Messages endpoints, and recommend setting additionalProperties to false on each object and listing every property as required, so the schema leaves no room for invented fields.

Per server, at launch. Per the same docs, --tool-strict-level takes three values:

auto (default): follow each request's tool choice and per-tool strict flags.
function: always constrain the call's outer shape (the markup and the tool name) for any request with tools, while leaving argument contents free unless a tool is strict.
parameter: pin the argument schema for every tool, as if each one had strict: true.

The flag came in through vLLM PR #56268, which describes it as server-side control matching SGLang's SGLANG_TOOL_STRICT_LEVEL. Ubicloud says the feature shipped as a server-side flag in vLLM 0.31; the latest release in vLLM's feed when this was written was v0.32.0, dated October 18.

The server floor is the one I'd reach for first. Agent frameworks generate tool definitions for you, and you often can't easily add strict: true to each one. A launch flag covers every client that connects.

Put this into practice

Plan for an afternoon, mostly spent measuring.

1. Check your version. On the serving machine:

pip show vllm | grep Version

If you're below 0.31, plan the upgrade first. The flag discussion assumes a current release.

2. Measure before you change anything. For Kimi K2 models, Moonshot AI publishes K2 Vendor Verifier, the benchmark Ubicloud used. Its schema accuracy is simple: the share of tool-call responses that pass schema validation. Ubicloud ran it as a 2,000-request workload. The evaluator reads a samples.jsonl file that isn't bundled in the repo root, so check the README for where to get the sample set. Then point it at your own endpoint:

python tool_calls_eval.py samples.jsonl \
    --model <your-served-model-name> \
    --base-url http://<your-host>:8000/v1 \
    --api-key <key-or-placeholder> \
    --concurrency 5 \
    --output results.jsonl \
    --summary summary.json

For other model families there's no ready-made equivalent, so log it yourself: count tool calls your framework fails to parse or validate, over a day of real traffic. That number is your baseline.

3. Turn on the server floor. A typical launch for a Kimi K2 model looks like this (the parser name comes from the vLLM docs):

vllm serve <model> \
    --enable-auto-tool-choice \
    --tool-call-parser kimi_k2 \
    --tool-strict-level parameter

--enable-auto-tool-choice lets the model decide when to call tools, and --tool-call-parser tells vLLM how to read that model's tool-call markup. Swap the parser for your model family. A simple rule: start with parameter, and if the benchmark shows worse output for a model, step down to function, which still fixes tool names and call shape.

4. Run the same measurement again. You're looking for schema accuracy near 100% with no drop in the other K2VV metric, trigger similarity (whether the model calls a tool when it should). A constraint should change how calls are written, not whether they happen.

5. Test speculative decoding separately. If you use it, run the benchmark with it on and off. Ubicloud links two open issues (#44006 and #55152) about wrong rejections in that combination.

6. Watch first-request latency. The docs warn of several seconds or more the first time a grammar compiles, so expect the first call with a new tool schema to be slow. Agents that generate schemas dynamically will feel this more than agents with a fixed toolset.

Honest limitations

A valid call is not a correct call. Constrained decoding guarantees the shape, so the tool name exists and the arguments type-check. It can't make the model choose the right tool, pass a sensible search query, or stop calling tools it doesn't need. Those failures still need evals.

Ubicloud's 100% is their result, on their models, on one benchmark. It's a strong signal, not a promise for your model. The post itself says support varies by model, and the bug list it links shows the edges are real. Issue #58930, for example, documents JSON Schema constraints such as multipleOf being silently dropped inside structural tags, so a "strict" call can still break a rule your schema declares. Keep your own validation on the tool side.

Constraints also depend on the parser. Per the docs, the generic hf response-template parser reads tool calls but rejects strict tools, so the floor only helps when your model's parser supports structural tags.

Some models write worse when constrained. If a model's natural tool-call format doesn't match vLLM's grammar for it, forcing the grammar can produce odd output. That's the reason the maintainers keep the default permissive. Measure per model, every time you upgrade either the model or vLLM.

There's also an environment variable, VLLM_ENFORCE_STRICT_TOOL_CALLING, that turns structural tags off entirely when set to false, per the docs. If constraints seem to do nothing, check that nobody set it in your deployment.

Finally, this is a self-hosting fix. If you call a hosted API, the provider controls this, and your lever is whatever "strict" option their API exposes.

The real lesson

When a self-hosted agent stumbles, the reflex is to blame the model and shop for a bigger one. Ubicloud's numbers point somewhere cheaper. Two calls in 25 were failing because the server let the model write anything, and one setting closed the gap.

Reading the docs closely turned up the sharper point: the default isn't wrong, it's cautious. vLLM hands you the choice because the right answer depends on your model. So make the choice on purpose. Measure your schema accuracy this week, flip the floor, and measure again. If the number doesn't move, you've learned your problem lives somewhere else, which is worth knowing too.

Sources: Ubicloud, "Fixing Tool-Call Accuracy in vLLM" (Oct 20, 2026); vLLM tool calling docs, stable (footer dated Oct 8, 2026); vLLM PR #56268; vLLM releases; MoonshotAI K2 Vendor Verifier.