Hacker News Reader

Comment by kmkarim | Hacker News Reader

Slightly off topic but is there recommended advice on how to tune / train not for document retrieval but for consistent JSON output with specific enums?

i.e given a text, always return back a certain set of fields. For some keys here is the possible set of enums etc. One shot prompting does work but curious how others approach this if you have training data on hand.

View on HN

There are many interesting tools that achieve this, like Outlines[0] and jsonformer[1]. I haven't tried them myself but they look very promising.

[0]: https://github.com/outlines-dev/outlines [1]: https://github.com/1rgs/jsonformer

You want grammars to restrict the output, search for "gbnf grammar". That and combined with a good prompt with an example, also check out outlines.dev

Ask the model nicely, check any json in the output against your schema, regenerate if it doesn’t match.

Crude, I know, but it’s compatible with every model. Which is useful if you want to compare the many different models out there.

That is a fairly good strategy. Six years ago at Capital One, I experimented with generating synthetic JSON AWS CloudWatch files for testing (that would not contain any sensitive information). Way back then I used LSTM models and I simply had code to check output and only keep valid samples.

LLMs are so much better for this than LSTMs now.

For OpenAI, use their functions schema mechanism.

Aside from that, take a look at llama.cpp grammars.

Microsoft Guidance will do this.

Seems Microsoft spun them out and gave them independence. Not sure why, given it's the kind of IP that helps keep microsoft dominant.

Reply on news.ycombinator.com