Hacker News Reader

Comment by pizza234 | Hacker News Reader

People still don't understand in 2026 that emergent properties can't be negated by the underlying mechanism. By that logic human thought is just electrochemical signals propagating through neural networks.

> it does not know

Define "knowledge" in a non-circular way.

There's a certain type of knowledge that machines definitely can't have - experiential - but conceptual knowledge doesn't inherently require subjective experience or a biological substrate.

> the only thing it does is reproduce patterns more or less abstract in function of the pressure you put on certain representation during training time.

As for humans; just replace "training" with "education" and the likelihood of a certain output with the likelihood of a certain thought or response.

Amusingly (in this context), much of adult behavior is shaped by patterns reinforced during childhood.

View on HN

As much as I flatly do not believe LLMs can have anything close to consciousness, I definitely agree that the argument of them being just a probabilistic word matching engine can also be applied to our own brains lol.

Agreed. It's quite hilarious that its basically biological probabilistic machine flexing on silicon probabilistic machine. On top of that that silicone one is right here.

> that the argument of them being just a probabilistic word matching engine can also be applied to our own brains

But that of course is only as it relates to language. Which isn't at all the primary function of the brain and driver of it's experience. It's the other way around because it's all of the other stuff that drives language. Considering the human brain a probabilistic word matching engine fails to capture the underlying forces that direct the language output because a human can say completely different things about the same scenario based on a huge number of non-language inputs or factors. Language is just what humans have come up with to communicate all of the other things.

> But that of course is only as it relates to language. Which isn't at all the primary function of the brain and driver of it's experience.

The same applies to LLMs: we know how they produce language, but the underlying processes that enable it may themselves constitute a form of intelligence.

Do you ever find yourself thinking about something deeply and then someone appears (it could be a child… for me it usually is) and asks you a question completely unconnected to whatever it was you were thinking about, and then - this is the bit that astonishes me - you find yourself answering the question verbally, maybe almost in full before you even re-focus your attention on the little person and whatever it was they asked ?

When I experience this it’s like the conceptual formulation and language expression part of my brain is quite happy to run itself for a few moments before I - the qualia of being me - has decided to experience that moment.

Utterly bonkers. I love it when it happens. It gets me thinking about how weird language and knowledge and consciousness really are.

> I definitely agree that the argument of them being just a probabilistic word matching engine can also be applied to our own brains lol.

You can apply anything you want. You can also apply the argument that our owns brains are just a block of cheese too, if you want. (Nobody will stop you.)

While one can quibble with its argument, Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge. The idea that knowledge can't be defined in a non-circular way is only plausible for someone with no familiarity with epistemology.

> Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge.

Ha. Gettier actually argued that justified true belief is insufficient for knowledge, not that machines can't possess it. And JTB doesn't inherently exclude machines.

Interestingly, research suggests LLMs encode truthfulness in their internal representations: https://aclanthology.org/2023.findings-emnlp.68/

Indeed. In fact this criticism of JTB is known as the Gettier problem: https://iep.utm.edu/gettier/

>Gettier's conception of knowledge as justified true belief (prob familiar to many philosophy undergraduates) implies that machines can't in fact accrue knowledge.

Nothing about the conception of knowledge as "justified true belief" implies that machines can't accrue knowledge.

For that to be the case "belief" would have to be defined in a circular way to exclude machines (which we can reject), or "justified" would have to be restricted to mean "based of observation of the external world".

The more general "based on observation of their input" applies to both men and LLMs.

That man can't know the bare direct facts of the world, just his own input (it's sensory experiences, and whatever second hand input he's given by other people), and for neither can he be sure they're absolutely true, or that he got right.

>The idea that knowledge can't be defined in a non-circular way is only plausible for someone with no familiarity with epistemology.

The idea that this or that epistemology restricts what can be the case in reality is only plausible for someone with no familiarity with epistemology.

> Nothing about the conception of knowledge as "justified true belief" implies that machines can't accrue knowledge.

They can't accrue knowledge because they stop learning once their training is finished.

By this definition if they could instead buffer recent facts and then offload to their weights reliably and efficiently they would be able to accrue knowledge despite their substrate.

Yes, and they'd have to deal with the same problem that we do: how do we know what is true? what gets counted as a "recent fact" and what gets dismissed as mere opinion or disinformation?

In training they get feedback. Out of training, not so much.

Hard problem

>Yes, and they'd have to deal with the same problem that we do: how do we know what is true?

So? We also have the exact same problem, and no solution that covers all facts, not even the most basic ones.

So them having this problem would be either neutral or supportive of the idea they're intelligent, not refuting it.

I was addressing the other half of the initial statement:

> Nothing about the conception of knowledge as "justified true belief" implies that machines can't accrue knowledge.

Even if the LLMs could adjust their weights in response to knowledge that they encountered post-training, they would need to make judgements about what is "true" so that they can only add "justified true beliefs" to their weights.

I agree that this is also a problem humans have, and we do really badly at it because we tend to weight social conformance more than factual accuracy.

And yes, I agree that this problem bears no relation to the question of whether they're intelligent, or even sentient. I was answering a different question.

You mean they don't accrue knowledge during the training?

Humans stop learning once their training is finished too, it's called death.

The difference is that LLMs we can talk to them when they're in that dead state.

And of course for an LLM this is an implementation detail. One could just as well keep updating the weights of the same model over time. And its being done, it's called "continual learning". LoRA is also a way of doing that.

That's an imposed limitation, nothing structural. The models do accrue knowledge during training so they already check that box. The LLM companies simply set it up such that newly gained knowledge can't be accrued after that, they could allow it just as easily.

As I understand it, the back-propogation of the weights adjustment involves comparing the result they got with the "correct" result from the data. The problem with data outside of training is that there's no "correct" result - nothing to judge against, so no method of back-propogating the weights corrections. But I could be wrong.

> Nothing about the conception of knowledge as "justified true belief" implies that machines can't accrue knowledge.

Interestingly, according to the parent's logic, there is a specific argument for that (took me a while to get the argument, and I think its not sound).

The reasoning is that, according to reliabilist responses to Gettier's problem, knowledge requires a reliable belief-forming process. But if we assume that LLMs operate probabilistically, they can't fulfill this, because they're "guessing".

The refutation to this argument is the usual one: levels of abstraction. LLM reasoning, like human intellect, can exhibit structured/logical behavior at a higher level of abstraction, even if the underlying processes are stochastic.

Therefore, if one bases the reasoning on the fact that both are stochastic at the lower level, then both can't produce knowledge, but if one bases it on the higher level, they both can.

(Having said that, I don't imply that AIs can reason like humans, but just that the criticism is not sound)

Gettier actually criticized justified true belief (JTB) as insufficient for knowledge and provided counter examples known as Gettier problems

Gettier cases are in fact the refutation of the simplistic JTB conception of knowledge.

> By that logic human thought is just electrochemical signals propagating through neural networks.

This argument keeps getting rehashed. The point is not the signal. The point is where/how the signal travels which in the human is affect by much more than whatever concept that can be represented by words.

I think the idea of emergent properties in LLMs is less and less relevant. Yes, the first few times they scaled pretraining and such models could do few-shot tasks was quite interesting. But the models we use now are really the product of extensive RL hillclimbing on a wide array of curated tasks. What’s interesting now is that the tasks they’ve been fitted to (long running tool call chains interleaved with reasoning tokens) do very useful things.

"emergent" is just a word people use when they can't explain how the properties of a system lead to its more complex behavior.

There's no magic involved, just as-yet-unexplained phenomena.

Everything in the chain of causality has some explanation or mechanism, understanding it step by step dissolves the illusion of "emergence".

>human thought is just electrochemical signals propagating through neural networks.

Is it not? Or are we discussing existence of a soul etc.

In all the centuries of human dissection, nobody has ever been able to find the soul particle inside a human. It appears that one does not exist.

I genuinely don’t know why people feel so certain about things like this. It’s like they mix up salience with reality.

I mean what can a person really be certain of, an experience is happening. I might be wrong about everything or confused. But I know I am having an experience. Do I know other people are? No. I assume they are.

Ask an llm what is truth that it is most certain is true. It will answer something similar. Do I believe it is conscious, no. But I also believe “I” exist, which is weird because where is the boundary between “I” and everything else.

So ok, we create meaning, not truth. Maybe just accept that.

There's no such thing as "emergent properties". The phrase is just a technobabble way of framing primitive animist superstition.

There are whole university degrees about "physics of complex systems". "Emergent properties" is a lens you use when looking at something. For example magnetism is seen as an emergent property of particles interacting and aligning. Thermodynamics can be seen as emergent properties of statistical mechanics (pressure, temperature etc). Agency is another example of lenses used to interpret something. Does a toaster have agency because it's trying to toast bread? Yes or no depending on the lenses you are using to describe its behaviour.

"Complex systems" and "emergent properties" are two absolutely different concepts that are absolutely not related in any way.

"Complex systems" are unstable systems that cannot be solved with simple equations. They do not magically take on properties that they didn't have before.

Brother you're arguing on HN of all places that education is a training pipeline while missing the fact that reddit and HN are the de-facto training run for institutional propaganda. How many reddit comments are just regurgitated knee jerks and puns? All of them that's how many. That's a training pipeline. How much of HN is braindead takes from people who value status more than knowledge and competence? All of it that's how much. That's another training pipeline for the next "Show HN: here's a picture of my cock". The fact that you recognize the "Show HN:" token is proof that you were trained. There's really nothing much to add here. I believe you already agree with me but in any case I spelled it out so that the next person can stop huffing the human chauvinism copium.

> There's a certain type of knowledge that machines definitely can't have - experiential

Can you justify this? Do you mean currently or ever?

I can certainly imagine a future architecture where I would describe a machine as capable of accruing knowledge through its experiences (read: experiential knowledge).

> By that logic human thought is just electrochemical signals propagating through neural networks.

My dear fried: the human being is much, much more than this. And deep down we both know it. Do not reduce us to materials y "randomness". Is your life a random coincidence? Not at all.

Okay, but then you can't do the same thing to LLMs, either.

> Is your life a random coincidence? Not at all.

Of course it is. That you do not like the idea of something does not in any way affect whether it's true. You'd like to feel special, but there is absolutely nothing special to us. We're just a random arrangement of atoms in the universe. That is objectively true, whether you like it or not.

That you were born at all is obviously completely random. Any prior event by any other party (butterfly effect) may as well have resulted in you never existing at all.

The only question is if basic physical effects are really random or deterministic, but that's irrelevant to us, there is no practical difference. In practice, it's all random.

And yes, our thoughts are just neurons firing as a result of our lifelong training on the inputs we were given (senses). It's all very complex, yes, but ultimately not fundamentally different from what a machine could do.

If you believe in unproven absurdities like god or immaterial souls, please be honest and upfront about it.

Imagine one day in the far future 21xx, we decide that machine actually has conscious since the 202x, and people in the future look back at us in horror: The kill(create new session) an "Android" without second thought, whenever they want.

Reply on news.ycombinator.com