jesdo

We currently focus on serving DeepSeek models on dedicated clusters for large customers. We do not offer this as a publicly available service at the moment, but we may consider doing so in the future.

I believe there is constrained decoding available for Qwen 3.8 in vLLM. But mileage may vary: although you guarantee syntactically-correct tool calls, you'll want to observe the overall quality of the model with and without constrained decoding. In the case of Kimi, there is a clear recommendation from the creators to turn constrained decoding on. I don't think this is the case for Qwen 3.8.

We are also interested in this and may focus on it in a future post. They are two different perspectives: (1) What is the overall quality of LLM+serving stack+harness, and (2) How close does the behavior of our setup match the first-party provider's? In this post, we focused on the latter. Moonshot AI, the creators of Kimi, recommend constrained decoding to be on by default, and also serve Kimi that way. This implies that it helps overall performance, though it does not provide numbers for it. We want to serve Kimi as recommended, and ran into the challenges described in the post.

There is work that suggests that constrained decoding can negatively affect model quality, but this depends on many factors, among which the model in question [1, 2, 3]. Overall, it seems to me that there is a trend towards using constrained decoding. OpenAI's newer Responses API has constrained decoding turned on by default as well, similar to Moonshot AI's Kimi.

1: https://arxiv.org/abs/2408.02442

2: https://blog.dottxt.ai/say-what-you-mean.html

3: https://dylancastillo.co/posts/say-what-you-mean-sometimes.h...

Overall, a fair assessment. I do want to place a footnote here that we did fix vLLM in the sense that its correct behavior for Kimi required code changes: it was impossible to fix it in the configuration. Later, the vLLM team fixed this problem upstream. Now, with the latest version of vLLM, the problem we describe can be addressed by setting the `--tool-strict-level` flag to `parameter`.

Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.

Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.

To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.

I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.

I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)

Great initiative, and good to see that it has an effect. I'm a bit sceptical about the available funds. 1 million over 5 years is a nice starting package (4+ PhD students), but the availability of overall research grant money in the Netherlands has been under pressure for years and is difficult to acquire. Researchers moving here may find it difficult to acquire further grant money compared to US, at least in CS.