Hacker News Reader

Comment by saurik | Hacker News Reader

Is the schema constraint not implemented in a way where the new distribution is identically scaled to the old distribution with retry? I had thought the way it worked was just to disallow invalid next states... oh, or is the idea that you think it is much preferred to retry at the level of the entire tool call, rather than (conceptually) per token? I guess I can see that making a lot of sense, as maybe the model gets "backed into a corner", so to speak, with a really bad initial choice and is then trying to rescue horrible states?

View on HN

The way it works is to disallow invalid next states, and getting backed into a corner is one way to think about one possible failure mode.

For a concrete example (happens less nowadays with bigger models, but it still happens, especially later into a context window), models hate long json strings. It's very easy to get a json output like

{"name": "Bob", "biography": "Bob is a fantastic builder, having ...

Critically, the ellipses terminate the JSON. With schema-constrained decoding, you _must_ eventually terminate that broken string, and it'll typically terminate it right after the ellipses. Basically, by the point you got to the first period the output was borked. You were backed into a corner. Retry as a strategy fixes that.

As another example failure mode that's a bit more subtle, you might have a set of "correct" tokens early on in the output which get the KV cache into a bit of a wonky state, causing it to have worse outputs toward the end of your output. Disallowing invalid next states forces you to generate tokens in that worse-output state. Retry lets you fail more often and then hopefully not have those wonky-state early tokens. It's not really that you were backed into a bad state as a hard rule, but that you were backed into a bad state as a probabilistic statement.

For a toy model to think about, suppose the entire grammar is [ab, ba], and the LLM's universe of training data is [ab, ab, ba, bb]. The "right" distribution (retry on failure) is {ab: 67%, ba: 33%}. The schema-constrained distribution is {ab: 50%, ba: 50%}.

What the other guy means is the model is forced to decode potentially out of distribution and subsequent parts of the sequence will be sampled from a higher-entropy distribution.

i.e. it might be that a particular model may need some additional conditioning, or perhaps adjusting the order of the json output affects it to produce lower-quality results.

You have to remember that the LLM will be expected to generate 10s of thousands of tokens on a single turn, where things like quantization noise lead to cascading error further on. So when you can, you really want to stay within the model's distribution.

I guess to be more concrete, I thought the way schema aware decoding worked should be identical to a token-level retry: "only even dice rolls are allowed" converting {0:0.1, 1:0.2, 2:0.2, 3:0.1, 4:0.2, 5:0.2} into {0:0.2, 2:0.4, 4:0.4}. That is why I am asking if the difference being stared at is doing the retry at the level of the entire tool call rather than each token (as this let's later errors in the schema re-roll the earlier committed states)? (But like, it isn't clear to me that's what they mean, so I feel a need to ask; as an example, you are now talking about field order, but that's an entirely different issue involving an overly-constrained constraint: nothing would prevent implementing the grammar for the constrained decoding to allow the fields in any order, and if you allow duplicate fields -- which JSON typically allows anyway -- then it is probably already supported as the language loses its additional context requirement.)

Schema-constrained is equivalent to token-level retry. It's not equivalent to retry of longer sequences.

Right... and I said exactly that in both my comments, but I was looking for confirmation that that's the issue being stared at, as it wasn't obvious to me that's what was meant and (I will cite as demonstration of the confusion) now people are bringing up other ideas (such as field order, which is an entirely unrelated constraint). (FWIW, since you are the person I was originally asking, I will take this as positive confirmation that "yes: I am saying it is better to retry the entire tool call" ;P.)

Ah, I see. IMO the other response [0] I gave is probably the closest to what I had in mind. Interestingly, retry very well might not be better, especially when measured on a pareto frontier including cost.

I have a reasonably strong prior that it should be better though. That comes from my observations that whenever I notice the difference in distribution the retry method is better (obvious sampling bias, but what kind of person ignores their own experiences). As a stronger piece of evidence and intuition, the entire goal of these systems is to reproduce a certain distribution. If that distribution says that you should use ripgrep 10x as often as you should use grep then that's a signal you probably don't want to throw away. Constrained decoding might very well do so and place them on even footing or skew 10x the other direction. Retry faithfully captures that original distribution. I'm skeptical of solutions which want to shy too far away from it (similarly with most applications of changing the model's temperature).

The more important point is just that they're different, so we should probably measure the impact.

[0] https://news.ycombinator.com/item?id=50245714

(Not sure you'll see this, ofc.) After percolating on this for a couple days, is retrying the tool calls enough? Maybe the reason the tool call is requiring three retries is because the AI simply got locked into too difficult of a way to solve the problem, and if it retried the entire response it would have gone in a different direction entirely. How far back do we feel comfortable?

These edge-cases are why the simplest answer works best: just scale your model to be good enough that it can work through the problem on its own

for putting together something like a tool-call decision model, the conditioning will take less time than getting the data for it, which also will not take much time if you keep your session data. whatever fails is your negative contrast, whatever passes is your positive data. the space of tool calls your agents will use in your particular environment is not actually that big, which makes this one of the easiest wins you can pull off.

> or perhaps adjusting the order of the json output affects it to produce lower-quality results.

The order of the output fields can actually be used to tune performance of the outputs. It can be helpful to force the LLM to reason about specific fields first if later fields benefit from seeing those earlier fields' output. You have to test it because some times the results are surprising in directions you didn't expect.

XGrammar has an option somewhere to allow fields to appear in any order but I don't know if vLLM exposes it. There are some tradeoffs and limitations for allowing fields to show up in any order because there is some complexity in schema parsing like that.

Technically true, but this is a very inefficient optimization strategy and usually indicates some structural issue with your setup.

When you know enough about the JSON structure to understand how to set a decoding order, then you can simply break the problem into smaller parts so it's more manageable.

Otherwise, you have these solutions: - GEPA-style prompt optimization for simple agents - If working with large models, you can just have them do step-by-step problem solving & relying on their CoT for accuracy. - A small model can simply be trained to do what you want

And if those don't work, I would be extremely surprised.

Reply on news.ycombinator.com