Hacker News Reader

Comment by orbital-decay | Hacker News Reader

Of course an LLM does have a latent concept of its own confidence that is internally consistent, and is perfectly able to articulate it, the author just chose the worst possible way to ask. Using a 0-100 scale for anything is a bad idea with both LLMs and people. Instead of rating it 0-100, split your problem into many dimensions/axes and give each a small and meaningful enum or just a less granular numerical scale (ideally just a binary choice). Then apply weighted scoring to the result. You'll be surprised how much better it becomes at meaningfully expressing its confidence this way.

YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.

View on HN

I would also like a source for this, I do not believe it is true and founded that "an LLM has a latent concept of its own confidence that is internally consistent".

LLMs generally cannot report the outputs of their own activation functions. Taking the output of softmax as if it were a probability score is considered a first-year PhD student misconception.

They can generally do a decent enough job with a likert scale and a rubric. Which isn’t quite the same thing as a confidence score and won’t be calibrated at all, but may be preferable to users.

Peoper confidence scores on a continuous scale, or even a fine grained discrete scale, are arguably skeuomorphic. It’s worth introspecting on whether that’s what you really needed in the first place, or if it’s just what you’re used to from past experience working with regression-type models.

Exactly my experience: given a set of 3-5 scores a and a sentence or two for each describing what earns that rating is quite effective in getting LLMs to score accurately. Failing to give a rubric doesn't give enough context, and using many values (like 1-100) produces more arbitrary results.

Yes. Though, also, I don't think it's correct to regard that as a confidence score (in the traditional sense) coming from the LLM. Most the value in a good rubric comes from how well the human prompt engineer can clearly and concretely describe the kinds of inputs that would support different confidence levels. Leaving it unspecified and letting the LLM sort it out doesn't lead to satisfactory results IME.

The upshot being, good "confidence" scores reflect good human analysis and prompt engineering, not some sort of introspective capacity on the part of the LLM.

> Of course an LLM does have a latent concept of its own confidence that is internally consistent, and is perfectly able to articulate it

Can you provide credible reference for this?

"LLM" literally converse/compute via statistical inference.

It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.

The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope and it's confidence about the context.

Anything else would be....original thought and exposition? I guess. You could say that our brains do the same, IE; unconcious statistical analysis leading to inference based on confidence.

> It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.

Does the output of the model have insight into those details? Just because our brains function based upon electrical potential doesn't mean we have innate insight into it.

Yes, this is what is user-level marketed as context memory.

Certain models of course force this context memory, some don't, some allow you to toggle it's "trigger level".

For example Anthropic, use haiku for low memory context queries, research. Then Opus/Fable for compiling/analysis/design, which relies on the LLM inference constantly going back into context memory.

Another example is OpenAI/GPT. The "free" tier forces context memory using the "highest available model" per your subscription (or lack thereof), and upon hitting its limit, drops you down to a low context model, and then ultimatley blocking that chat entirely.

(I appreciate this discussion and am not trying to inform you, I'm sure you already know, but typing these things out helps me understand my holes in understanding.)

Excuse my ignorance, but how does context memory provide an insight into a model's inner workings? I mean this:

> It is given sets of paramaters/data and infers a response based on it's acceptable confidence level. The bottleneck for this is compute power and model/prompt/rails/integrations.

or whatever the LLM claims its confidence is on any given output?

You converse through electrical signals and muscle contractions, but that will not make you an expert in either.

Similarly, a language model trained to emulate speech, follow instructions, and solve problems will not implicitly have an understanding of its own statistics. The research doesn't back that up and it just doesn't make any sense if you think about it.

The only thing LLM produce that could be considered its "confidence" would be its token logit values, but even that is a sketchy connection.

I didn't say anything about expertise, I do not use LLM for work and have a background in statistics/teaching. Also, none of this is related to tokenization. It's just the way the application layer infers response.

Any inference or probability engine, be it an app or a heuristic mental process, is obviously based on categorizing historical data and literally weighing it's confidence around it to form responses.

Confidence intervals and associated indexes/logic are the foundation for most heuristic data models.

> The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope

"It's aware" is a big leap and bold original thought, as awareness is not something it has full stop, try different terminology?

This is an incorrect conclusion. Your statement about statistical inference is correct but it doesn't explain why your initial statement, especially "is perfectly able to articulate it".

A good litmus test for this is, typically any artificial neural network, perceptron or even a simple linear regression model do statistical inference based on weights and selection based on confidence level. It's a generic characteristic, which doesn't explain anything specific to LLMs.

> The reason LLM stays consistent/can be somewhat trusted to stay on rails is that it's aware of both it's context/scope and it's confidence about the context.

Post hoc ergo propter hoc.

> I guess. You could say that our brains do the same

I understand that it "feels" the same from a certain perspective, but we don't know that.

Lets be honest with Youtube's system: it's not about what people really want; it's about whats enough to make them _feel_ in control.

What many people would want it just a complete block on select youtube channels and providers. Just whole hog, rip them out of any of the potential recommendations.

But that's not whats best for Youtube. The point system still stands in ratings, once exposed to the viewer, are gamified. They would however still work if it was a secret vote, but then, again, the user wouldn't be compelled to use it as people want immediate visual cues that they're doing what benefits themselves.

> What many people would want it just a complete block on select YouTube channels

This exists, doesn't it? "Block channel" (or rather, "Don't recommend channel") is probably the feature I use most... I must have thousands of thousands of channels on that list.

The alternative is "Not Interested", but at least on AppleTV you're forced to then choose either "I've already watched the video" or "I don't like the video", and I have literally no idea what to do with those choices. If I've already watched the video then the reason I'm saying I'm not interested is because I didn't like it, so both are true, but I can't select both; and if I just don't like the clickbait thumbnail or title I haven't watched it and I don't know if I like the video itself, so neither is true, but I can't select neither...

Every time I think about YouTube's interface it becomes more obvious it's made by people (?) who's brains just don't work like mine...

Do you have any evidence whatsoever that that score is accurate?

LLMs have trouble with open labeling tasks, this was studied. But, what also was studied is that if you instruct it how to score something, it can do it not so far than a human. If you leave out the instructions, I believe it can be considered an open codebook and therefor it probably wont work well.

> Using a 0-100 scale for anything is a bad idea

Even a probability?

If it is nominally a probability then make sure it’s well calibrated.

If it is something like cosine similarity or SVM scores, which aren’t probabilities or even percentages, then users will misinterpret it if you present it as a number from 0-100 or 0.0-1.0.

That's very well put. I noticed the issue myself when building my own evals and naturally started drifting into the direction you are describing, but you describe it so succinctly that now I am seeing how it should actually be done. Thank you

Yeah, I agree this approach can produce usable results. The more narrow and well scoped you can make each decision the model has to make (e.g. binary vs 0 to 100) the better the results you'll get (and the easier it is to troubleshoot and tune).

When renting Airbnbs, owners frequently put explainers in the rooms what each of the star ratings is supposed to represent. Similar problem.

Can you provide an example f how you did that? Very promising!

>>YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.

As always, I'm in a minority - I have a VERY clear idea what to do with it :-).

(1/2 - never ever show me crap like this again. 3 - decent mainstream stuff. 4 - This meets my interests, is good, and I'd like more of it please. 5 - this is endorphin analog candy for me. Scott Pilgrim, Hunt for Red October, Neuromancer, Dune grade content :)

Then they ushered in an age of garbage by removing the scale altogether.

The scale is almost never useful. You cannot quantify most things. Even with humans you get inconsistent results and confusion. As the parent said, a binary (ok/not ok) or ternary (bad/ok/good) choice is what’s appropriate, most of the time. If this is not granular enough, the vast majority of the time it means that you need more axes.

Reply on news.ycombinator.com