Comment by Volundr | Hacker News Reader

Comment by Volundr | Hacker News Reader

Parent Full thread

> What is the practical difference you're imagining between "consistently correct guess" and "knowledge"?

Knowing it's correct. You've just instructed it not to guess remember? With practice people can get really good at guessing all sorts of things.

I think people have a serious misunderstanding about how these things work. They don't have their training set sitting around for reference. They are usually guessing. Most of the time with enough consistency that it seems like they "know'. Then when they get it wrong we call it "hallucinations". But instructing then not to guess means suddenly they can't answer much. There no guessing vs not with an LLM, it's all the same statistical process, the difference is just if it gives the right answer or not.

> Knowing it's correct.

I love the convergence with philosophy here.

This is the second reply that's naively just asserted a tautology. You can't define "knowledge" in terms of "knowing" in the sense of the English words; they're the same word! (You can, I guess, if you're willing to write a thesis introducing the specifics of your jargon.)

In point of fact LLMs "know" that they're right, because if they didn't "know" that they wouldn't have told you what they know. Which, we all agree, they do know, right? They give answers that are correct. Usually.

Except when they're wrong. But that's the thing: define "when they're wrong" in a way rigorous enough to permit an engineering solution. But you really can't, for the same reason that you can't prevent us yahoos on the internet from being wrong all the time too.

Maybe LLM's know so much that it makes it difficult to feel the absence. When someone asks me about the history of the Ethiopian region, I can at most recall very few pieces of information, and critically, there is an absence of feelings of familiarity. In memory research, familiarity signal is can prompt continued retrieval attempts, but importantly absence of that signal can suggest that further retrieval attempts would be fruitless, and that you know that you don't know. Maybe knowing so much means that there is near saturation for stop tokens...or that llms need to produce a familiarity like scoring of the key response of interest.

Reply on news.ycombinator.com