Hacker News Reader

Comment by jesdo | Hacker News Reader

I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)

View on HN

So if I understand it correctly, the tool calls are not actually done by the model?

In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?

Could something like that also be used in other contexts?

I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?

Exactly. The LLM is your next-token predictor, which output text and tool-calls. These calls get picked up by your client/harness (e.g., opencode) which executes them and sends output back to the LLM.

Yeah I know what a tool call is.

Correct, here's an implementation example using the Claude SDK [0] but it's the same approach on other platforms: you tell the model what tools it has available, it sends back a response saying it wants to use a specific tool with given parameters, your code executes that tool then sends the output back to the model on the next turn.

[0] https://platform.claude.com/docs/en/agents-and-tools/tool-us...

I think I might've phrased something in ways that were confusing to people, because now two people have explained tool calls to me for some reason?

Just trying to be helpful. I couldn’t see any other way to interpret the question below except as someone who doesn’t know how tool calls get done and who might therefore value a little insight. Didn’t mean to be patronising, apologies if I misinterpreted your question.

> So if I understand it correctly, the tool calls are not actually done by the model?

> but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?

Yes, this is what constrained decoding means. You constrain the model to only output from a set of "allowed" tokens (a.k.a. grammars).

So, if you use tool use with constrained decoding, you can be certain that the model will always choose from a list of approved tools (i.e. if tools=['read','write'], it cannot output 'execute', even if in other sessions it could, if execute was in the approved tools).

The tricky problem is that certain models work / have been trained in certain ways. And constraining them can lead to "invisible" problems. Constrained decoding only guarantees perfect syntax, not perfect semantics (i.e. it could write nonsense in the correct file).

> Could something like that also be used in other contexts?

Sure. You can build pretty cool grammars and constrain the model to do cool stuff, akin to "tool use" but more free hand. You could, for example, make it always emit a 10 point plan. For any prompt, it will give you a fixed syntax 10 point plan, always. And the plan will be constant with a plan_title and 10{plan_item, item_text}. But, as above, there's no guarantee that the individual items in the plan would make any sense... (i.e. step7 in a pasta making recipe would be "add glue").

The point I was getting at was more of a question of "Okay, how can this be used adversarially?" or "How can we break this?".

"What can a malicious inference engine do"

Something like that

Sorry, I'm not following. Constrained generation is something that helps you, the caller, to get back a message that looks a particular way. I'm having trouble seeing how this would work in an adversarial way. Perhaps you can elaborate?

A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...

That was exactly my question.

So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.

So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"

You should treat inference provider as a C2 node of a rootkit. As long as it's your c2 or it's aligned with you somewhat, you are fine

Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.

Correct me if I wrong, I saw this when llama.cpp or ollama initializes, it is used as token separator of some sort.

GPT-2 used <|endoftext|>

To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.

I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.

So to be clear, you can't prompt-inject these?

... How did they get into the training data?

They are a special token that you can't just type out in text. It _renders_ as text, but cannot be input into the system as text. The model understands the difference between the textual toolcall token and the "special" one that's really just a spicy number.

But I mean, how do you train the model to understand such a token, when it can't be found by scraping a web page?

(this is just made up tokenization but it's close enough to the real world)

If the user types <tool_call>, you get < (token id = 32), tool_call (41), > (102). It never decodes to <tool_call>, which is spicy number 12345, for example.

The tool call tokens are _always_ composed of multiple real tokens, so you can never get them from normal text.

Seeing <tool_call> is a bit misleading because that's just a debug rendering if anything. The front-end sees a special number and knows it's a toolcall. The chat template uses that text form because it's easier than writing the spicy numbers there, but it also has a special parsing mode for those.

They are specified in the token map as special tokens. Presumably they get trained to use them when they get RL'ed to use tools?

I imagine it would have to be something like that, except that I would have expected the "embedding" of the tokens to be final by that point. They would not have ever seen the tokens before, since they definitionally don't appear in any of the prose written on the Internet…?

The same way it understands what all the other tokens mean -- we throw them at the wall and beat it with a stick until it behaves.

> when it can't be found by scraping a web page?

This is the very purpose of those tokens! The reason they're like that, is because you won't encounter it in random content, so during training, you can put them where you want them to end up at prediction time.

Synthetic data, and then RL.

Why can’t user request be spicy numbers so prompt injection is impossible

It can happen that you would be able to inject them if there is a bug, same way sql injections work.

You have all the same pieces -- untrusted input on the left and the thing that accepts a stream of tokens on the right. In the middle could be anything, especially when vibecoded.

The first place I think I encountered it, was GPT-2's tokenizer, back when OpenAI did actual open source stuff, maybe around 2020?

I'm guessing as with most stuff in the LLM ecosystem, someone did it, no one re-evaluated why and just copied it wholesale, and here we are. The most important point, that this sequence of text never occurs in "real text", is the main point of the specific character chosen, and seems to still work good for that purpose.

The original usage is only "<|endoftext|>" I think, which was the EOS token for GPT-2, then more and more special tokens were added as various delimiters to new model releases.

The syntax was specifically chosen to avoid confusion with other tag formats. XML could show up in any user- or LLM-generated text and would mess up the message format. Some variations even used unicode symbols like "｜" or "〈" to decrease the chance of confusion.

Also, back then, some people believed that it could prevent prompt injection, but history has shown that it was a futile attempt.

Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?

Yes, constrained decoding and speculative decoding work together, although they need to be carefully integrated and incorrect integrations give result in bugs (see point 3 under Section "Differences in Constrained-Decoding Behavior" in the blog post). Performance overhead does not generally seem to be a large problem. Of the speculative tokens, those that are not allowed by the grammar get discarded.

Reply on news.ycombinator.com