Fixing Tool-Call Accuracy in vLLM
ubicloud.com
ubicloud.com
The blog post is rather confusing because until you understand the entire situation it appears to imply that they fixed vLLM (what the title implies) but really they just needed to configure it correctly.
To be fair, the course of constrained tool calling configuration in vLLM changed over time in ways that changed behavior, which is the real problem. The Ubicloud team deserves a lot of credit for having benchmarks in place to track performance and catch regressions for investigations. Some of the providers you find on OpenRouter have inexplicably bad performance relative to others and clearly do not care this much.
However, if this article left you confused because constrained decoding and constrained tool call schema have been features in vLLM for a long time, jump to the end where they show what configuration changes they made.
Just as a point of feedback for the authors, I enjoyed the article, and did not suffer the confusion pointed out above.
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
Wait, can you actually do that? The model should not exactly care where in the context the tools are, but I haven't realised it's a thing that works
For example, if you want to use an LLM API that does not natively support tool calls, you can tell the LLM to emit the tool calls for your preferred tools in a markdown code block and parse them yourself.
I was wondering how feasible it is to modify the context, instead of just appending - e.g. to remove malformed tool calls and their error messages that would pollute the context otherwise.
Sure, you'd have to rebuild the KV cache and take a performance loss, but that might be acceptable if it happens rarely enough.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
... How did they get into the training data?
If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.
The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.
Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.
This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.
You have all the same pieces -- untrusted input on the left and the thing that accepts a stream of tokens on the right. In the middle could be anything, especially when vibecoded.
I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.
The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.
Also, back then, some people believed that it could prevent prompt injection, but history has shown that it was a futile attempt.
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?
[0] https://platform.claude.com/docs/en/agents-and-tools/tool-us...
> So if I understand it correctly, the tool calls are not actually done by the model?
Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).
So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).
The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).
> Could something like that also be used in other contexts?
Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").
"What can a malicious inference engine do"
Something like that
A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...
So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.
So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"
i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.
You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.
I have a reasonably strong prior that it should be better though. That comes from my observations that whenever I notice the difference in distribution the retry method is better (obvious sampling bias, but what kind of person ignores their own experiences). As a stronger piece of evidence and intuition, the entire goal of these systems is to reproduce a certain distribution. If that distribution says that you should use ripgrep 10x as often as you should use grep then that's a signal you probably don't want to throw away. Constrained decoding might very well do so and place them on even footing or skew 10x the other direction. Retry faithfully captures that original distribution. I'm skeptical of solutions which want to shy too far away from it (similarly with most applications of changing the model's temperature).
The more important point is just that they're different, so we should probably measure the impact.
The order of the output fields can actually be used to tune performance of the outputs. It can be helpful to force the LLM to reason about specific fields first if later fields benefit from seeing those earlier fields' output. You have to test it because some times the results are surprising in directions you didn't expect.
XGrammar has an option somewhere to allow fields to appear in any order but I don't know if vLLM exposes it. There are some tradeoffs and limitations for allowing fields to show up in any order because there is some complexity in schema parsing like that.
When you know enough about the JSON structure to understand how to set a decoding order, then you can simply break the problem into smaller parts so it's more manageable.
Otherwise, you have these solutions: - GEPA-style prompt optimization for simple agents - If working with large models, you can just have them do step-by-step problem solving & relying on their CoT for accuracy. - A small model can simply be trained to do what you want
And if those don't work, I would be extremely surprised.
For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like
{"name": "Bob", "biography": "Bob is a fantastic builder, having ...
Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that.
As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement.
For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}.
There is work that suggests that constrained decoding can negatively affect model quality, but this depends on many factors, among which the model in question [1, 2, 3]. Overall, it seems to me that there is a trend towards using constrained decoding. OpenAI's newer Responses API has constrained decoding turned on by default as well, similar to Moonshot AI's Kimi.
1: https://arxiv.org/abs/2408.02442
2: https://blog.dottxt.ai/say-what-you-mean.html
3: https://dylancastillo.co/posts/say-what-you-mean-sometimes.h...
Things get more interesting if you don’t give it a call syntax and just see what it does based on the name alone.
100% success is critical for tool calling.
The probability that the overall agent iteration is successful would scale very poorly even if we achieved 95%. With p95, if you need 50 tool calls that must execute sequentially to achieve a result, the chances that you won't have an error are only ~8%. There are ways to recover from bad tool calls (error feedback helps a lot), but if the agent is persistently using the wrong tools with the wrong args the overall system will not work very well.
https://github.com/huggingface/transformers.js/blob/main/pac...
I do things that require ~300 tool calls per prompt and I have been stuck with OpenAI and Anthropic since 2024.
If people still wonder what moat OpenAI and Anthropic have: They got 100% tool-calling accuracy (using constrained decoding) and prompt caching right all the way back in 2024.
I have multiples of that with some regularity, what is the hangup?
Lots of code "typos" aren't immediately syntactic errors. Misbalanced parentheses, for example, only bite when one needs to close the series. The solution is to go back and fix the original error, but that option is foreclosed to autoregressive LLMs. In the meantime, forcing the model off what it "wants" to generate might make the remaining syntactically valid tokens semantically meaningless, impairing future generation in a way that's hard to quntify.
Constrained generation for tool calls is certainly a type of syntax checking, but it's one that can be justified because the "hard" syntax choices come up front, both in the very format of the tool call itself to the name of the tool and its parameters. Errors are more likely to be detected in the same root-cause tokens that generate them.
[0] - https://github.com/ggml-org/llama.cpp/tree/master/grammars [1] - https://github.com/ggml-org/llama.cpp/blob/master/grammars/c... [2] - https://gist.github.com/HanClinto/b853571136fd42f8e2bde4c99c...