Typesafe claims that Jev is calibrated, but there are plenty of examples where it completely fails (predicting die roll being the most obvious one).
Unfortunately calibration is hard to benchmark.
If you instead give it a list of probability for each number and ask it whats the probability of each number, the result will be accurate.
Did i hear that correctly? In order for Jev to be accurate you have to give it the answer before asking for the answer?
(btw this is exactly how Jev is playing games).
What am I missing?
As in, neither LLMs nor Jev are truth engines. Truth comes from the provided context.. Plus weights.. kind of fuzzy, sorry I’m thinking out “loud”
This is not a “system one” question.
You’d be much better off using a proper LLM here: more accurate, can provide justification, can be steered when mistakes are made. Jev is going to be a coin flipper with almost no knobs.