Isn’t this what Microsoft Guidance does?
For example, suppose that you specify this as your Guidance "program" and suppose (for sake of simplicity) that the token for "lea" is 1300, the token for "ther" is 1500, and the token for "ves" is 5300:
"armor": "{{#select 'armor'}}leather{{or}}leaves{{/select}}",
Guidance will send OpenAI a chat completion starting with "armor": "
... providing a logit_bias map {"1300": "100"}. This bias forces the model to choose "lea" as the next token. Following this call, we have the prefix "armor": "lea
... and now Guidance calls chat completion again setting the logit_bias map to {"1500": "100", "5300": "100"} to indicate that the tokens for "ther" or "ves" are equally probable and really the only tokens the model is allowed to select between, unless some other token is maximally probable given the context. OpenAI now replies with token "1500" (let's say) and Guidance completes the string as follows: "armor": "leather
... because "ther" is represented by token number 1500. Guidance then tacks on the closing quote and other stuff specified by the user: "armor": "leather",
... and it sets the value of "armor" to "leather" so that you can use that value later in your code if you wish to. Guidance is pretty powerful, but I find the grammar hard to work with. I think the idea of being able to upload a bit of code or a context-free grammar to guide the model is super smart.https://github.com/microsoft/guidance/blob/d2c5e3cbb730e337b...