Hacker News Reader

Comment by RugnirViking | Hacker News Reader

Have you tried asking jev to predict the result of a con flip? My one and only prompt to jev so far has been that. After I 90% heads as a result, I figured I'd wait a bit before checking again to see if this tech has become useful

View on HN

You'll get 50% prediction accuracy for a fair coin no matter what result a model gives. Try testing adversarial games. Mixed strategies make actual sense in them.

The model gives a probability distribution as an answer; you can check to see if two arbitrarily chosen probability distributions are similar or not.

I mean predicting the result of a fair coin toss is not a classification task those models are trained to do. It can be framed as a decision making task, but for a fair coin any probability distribution of heads/tails will do equally well.

BTW, for a task of predicting a 51/49 biased coin, the correct answer is 100/0.

Not if you use a proper scoring rule.

See: https://en.wikipedia.org/wiki/Scoring_rule

I'd say selecting a scoring rule that depends on the probability distribution (the Brier score, for example) changes the task from predicting the results of coin flips to predicting the distribution. Maybe it will work with a corresponding prompt.

You can use a local proper scoring rule.

I'm dying to know if you got heads or tails when you flipped the coin.

That's like not using an LLM for coding assistance because it can't count r's in strawberry.

Try it yourself to prove you’re useful. Predict my coin flip.

I can't, because I don't think you have any coins on you. :)

Okay: { heads: 0.5, tails: 0.5 }

50% heads, 50% tails. What would you have given for predicting my coin flip?

Sorry, I underspecified. Your prediction must be one of either HEADS or TAILS. No other prediction is acceptable.

Thats not how jev works? We're talking about a decision model that gives a number next to a series of options that, taken together, add up to 1. There is some sensible debate here about whether they should be taken as probabilities, but I can't follow what you're getting at if you did actually understand how jev works

I did take your saying that it gave 90% heads to mean that it picked heads 90% of the time, not that it gave a 90% weight to heads.

I’d still maybe ask it a few more things before concluding that it’s useless. It’s meant to predict plausible responses and provide confidence scores, not compute real-world probabilities of events.

A probabilistic system that cannot reliably reproduce probabilities - somewhat ironic, but absolutely not its purpose.

It’s not really ironic since the goal of the system is to reduce variance. Or I guess ironic in the same way as it is that the Buckingham Palace guards just stand there when they could leave at any time.

No matter what answer is given, the prediction success rate is 50%. There’s no difference between the model giving a head answer 90% of the time and 50% of the time.

Try asking it to write a program to predict coin flips. That's what it is trained to do.

Jev literally cannot write programs.

Well, you can always ask it to predict a next bit in the code. Or even a letter. Maybe combine multiple letter into one and call it a token. We'll give a name to a procedure that gets tokens - decode. To improve speed, we can predict multiple tokens speculatively. To improve quality we'll give it extra space to store Chain of Thought. Ah, wait, we just invented LLMs

My bad, maybe I read it wrong or the comment was edited, but I was referring to LLMs in general. You won't get a fair coin flip from the model either way.

Found the human in the comments section.

this is not what *decision models* do.

Reply on news.ycombinator.com