Hacker News Reader

Comment by jncfhnb | Hacker News Reader

… why would you have the LLM spit out a json rather than define the json yourself and have the LLM supply values?

If the LLM doesn't output data that conforms to a schema, you can't reliably parse it, so you're back to square one.

It’s significantly easier to output an integer than a JSON with a key value structure where the value is an integer and everything else is exactly as desired

That's because you've dumbed down the problem. If it was just about outputting one integer, there would be nothing to discuss. Now add a bunch more fields, add some nesting and other constraints into it...

The more complexity you add the less likely the LLM is to give you a valid response in one shot. It’s still going to be easier to get the LLM to supply values to a fixed scheme than to get the LLM to give the answers and the scheme

Is there a general model that got fine tuned on these json schema/output pairs?

Seems like it would be universally useful.

How would I do this reliably? Eg give me 10 different values, all in one prompt for performance reasons?

Might not need JSON but whatever format it outputs, it needs to be reliable.

Don’t do it all in one prompt.

Right, but now I’m basically running a huge performance hit, need to parallelize my queries etc.

I was parsing a document recently, 10-ish questions for 1 document, would make things expensive.

Might be what’s needed but not ideal.

LLM performance is a function of the number of tokens, not queries

Reply on news.ycombinator.com