Comment by lukev | Hacker News Reader

Comment by lukev | Hacker News Reader

Parent Full thread

Upon a second reading, this is an excellent point.

For the sake of clarity, let's remove LLMs from the equation and posit the existence of Encyclopedia Eric. Ask Eric any question, and he will happily research it and come back to you with the answer. But he can sometimes be sloppy in his research, and he gives the correct answer only X percent of the time.

Furthermore, Encyclopedia Eric steadfastly refuses to cite his own sources or explain his reasoning in any way. He simply states his answer.

Can Eric be a source of knowledge? It seems evident that the answer is no, for low values of X. For higher values of X, the question becomes murkier.

The temptation at this point is to give up on defining knowledge at all and fall back on a sort of Bayesian epistemology where everything is ultimately a matter of probabilities.

Yet there does seem to be a distinct practical difference between a knowledge source that is "traversable" (like a standard encyclopedia) vs a knowledge source that is not (like Eric.) Is that part of the definition of knowledge? You're right, that is at least a Gettier adjacent question.

I think we can all agree that for current LLMs the value of X is definitely too small to count as knowledge.

I am glad that you think it an excellent point!

I think that this might be a nice way of getting round my objection, but there is one worry, which is that X is relative to a distribution on the questions we ask when we aren’t dealing with Encyclopedia Eric but with an LLM. I don’t actually use LLMs very much myself, partly out of arrogance and Luddite tendencies. But I suspect that the value of X for some sorts of questions (simple quiz questions, maybe?) and some LLMs (maybe not Google’s) will be high enough to end up in the murky case.

Of course, both you and I agree that there /is/ clearly a difference. I can see the attraction of appealing to the intuitive or pretheoretic notion of knowledge, since it’s a fairly straightforward way of stating the difference and it’s not obvious how else one might put it (I suppose ‘LLMs don’t explicitly think through the facts stored when deciding what to say’ is one way of putting it.)

I remember some time ago rather sleepily watching John Hawthorne talk about conditionals; my sole memory was of his banging on about ‘the little logician in the brain’ (I think the point was something like: some conditionals seem [in]felicitous in virtue of form because the little logician in the brain is reading them; others seem infelicitous because we examine them more closely and look e.g. at the referents involved, in which case e.g. Gricean considerations apply). One difference at least in the case of LLMs that makes sense to me is that there is no ‘little logician’ in LLMs.

> X is relative to a distribution on the questions we ask when we aren’t dealing with Encyclopedia Eric but with an LLM.

Assuming I understand what you mean here correctly, this should be the case for both LLMs and Encyclopedia Eric - there are topics Eric knows by heart (or thinks they know); there are specific phrases seared into his mind through sheer exposure during his life prior to becoming a living Encyclopedia. There are words he's used to, and exact synonyms he barely recognizes. All that means your chance of getting correct answer to your query depends, in complex and unknown to you way, on how you state it.

I think the thought experiment can be set up both ways. In one case Eric has a fixed probability of getting /any/ query right. (This might leave boosting open so we might want to gerrymander repeated queries out.) In the other this is relative to the distribution of queries.

A relevant point to this is the notion of "System-1" vs "System-2" thinking. Somewhat dubious when applied to actual human psychology but I think a valid metaphor for how LLMs work: they are only capable of System-1 thinking; a single forward pass through the weights of intuition

In my actual life, I don't trust my own System-1 thoughts: for anything important, I'm always going to engage System-2. And LLMs don't have a System-2.

(I also agree that when dealing with LLMs the value of X is not a single value but a highly complex space depending on the nature of the question and the training data of the model. In my mind it does not change the epistemological equation, it just means that even the value of X itself is harder to "know", so this ambiguity can only ever make LLMs a less viable source of knowledge.)

I am not sure that the ambiguity has to work that way. The suggestion I am making is that if we fix the distribution on questions and the training data, we might (a) know the value of X in this specific case, (b) be able to ensure that it is fairly high.

I’d say this is the murky case because on the fixed training data and query distribution X ≈ 1 and we know that even though we don’t know the value of X on other training data and other query distributions. I think that might be where the disagreement lies.

Reply on news.ycombinator.com