It's parasitic on humans and it's not an infinitely deep well. Wrong?
So Chomsky thought of himself as a philosopher of science more than anything, and his life mission is/was to answer a fascinating question: what kind of creatures are we? There's been a growing consensus going back to at least Darwin that we're separated from the rest of Animalia by our "reason" (!= intuition != understanding != judgement != volition), but he helped popularize an old theory[1] that puts it kind of beautifully: human language is only possible in beings capable of producing an infinite set of meaningful outputs in response to a finite input.[2] And so far, that list includes only two definite members: Homo sapiens sapiens, and computer programs built upon large language models.[3]
So all that to say: yes, I think you're wrong; the programmatic ensembles that brought us today's breakthroughs are an "infinitely deep well". As long as we have questions to ask, they are not theoretically constrained from answering them.
Of course, this invites some obvious follow-ups on practicality; even the most impassioned ASI boomer wouldn't argue that the exact ensembles used here could answer every question in the universe. But A) that's possible because some infinities are bigger than others, and B) the exact same caveat applies to humans, whose cognitive capabilities are thoroughly determined by accidents of biology, culture, and episteme. The latter is the thrust of Chomsky's famous debate with Foucault, which is absolutely worth a watch for anyone who hasn't seen it (starting around ~18:00, AFAIR): https://youtu.be/3wfNl2L0Gf8?t=1080
---
TL;DR: I ultimately disagree on both points, yes. In the ways that matter, machines are no more constrained than animals.
[1]: https://en.wikipedia.org/wiki/Wilhelm_von_Humboldt
[2]: Many animals are capable of communication, often quite complex and fascinating communication at that; even flora and fungi communicate! But language is qualitatively distinct. You may have heard this in the pithy-yet-mindbending sentiment "we've taught many apes to communicate with sign language, but none of them have ever asked a question."
[3]: It's no surprise in hindsight that trying to build better autocorrect got us to AGI; I find it unspeakably tragic that Chomsky never got the chance to see how thoroughly he's been vindicated, in this regard. He had his stroke mere months into the singularity, and had a quite negative view of the approach before then.
In LLMs that's a bounded infinity though, the models themselves have only so many bits and the activations are also limited. Once you reach the same state twice the model is looping and it won't break out of that loop. I've seen such loops across tens of thousands of tokens so I'm not sure to what degree the infinite output rule holds true. It would be an interesting experiment to try to figure out to what extent the 'busy beaver' can be kept busy before it degenerates to looping or nonsense.
He may just be parroting something he heard.
Language to humans is a symbolic mapping of labels to reality based on actual experience.
"Language" to LLMs is just a string of tokens to manipulate based on relationships harvested from recorded human experience.
It doesn't mean anything other than 1s and 0s - it's not a language because the LLM isn't communicating with you, just as the sum of humanity isn't communicating with you.
It's just a really good "choose your own adventure" book.
Humans aren't magical.
They also aren't a glorified library. They are authors, they generate the books in the library.
Cross reference a load of books and you will get useful relational info, but it's still a library index in a library.
An LLMs experience is everyone else's subjective view of their history. It doesn't make its own and it never can.
An LLM is a clone of Guy Pearce trapped in a context window which is just reruns of memento with script variations.
Humans may not have many original thoughts, but the thoughts that they arrive at are the product of (at least partially) independent processes.
An LLM is constructed. It no more had its own thoughts than a physical book has written its own story.
We're just going one step back. Instead of picking up a book, we're picking up a book generator. A book generator that was created from scanning in a bunch of books. All those books have authors and the book generator has essentially been authored by that collection of authors.
The book generator can very convincingly tell me what it is like to be a parent, in every way, but all it's doing is regurgitating data. It's never been a parent, it will never be a parent, it has never held a child or even experienced a single emotion. The word "parent" is just a collection of tokens. It's not a label that it uses to make sense of its experience, or other llms experience, of reality.
It's just an overly elaborate chinese room.
Yes and what you are saying is sense organs? To an llm twxt input is a sense organ, these days they can also read images and videos. I am not very sure what it means to not feel something, pregnancy like any experience or memory is a data, most of it should be in our brain while some of it would be interactions of the brain with the rest of the body. Even independently of this, there is no reason to think something trained on human written text from the human pov won't in its raw form (before rlhf.or whatver) think itslef as human and experience humna or human like emotions.
The collection of tokens argument sounds strange to me, do you think brains don't store data as electrical and chemical states?
Humans have subjective experiences, they create individual memories, they relate these using language.
LLMs are trained on accounts of the experiences of others to regurgitate a facimilie of that experience.
As a human, my input data isn't a diary account or a book or a home video. It's a real world experience that I participate in, and that I use language to communicate.
An LLM is a facimilie of everything it's made up of. It doesn't have its own agency, it doesn't have its own thoughts. I can ask it what it is like to ride a roller coaster and it will give me some text, but what is that text? It's something that someone else wrote about something the llm has no real experience of.
The llm is in platos cave staring at other peoples shadows.
The fact that an LLM doesn't have agency is a matter of how we present them, there is no reason they can't be presented to run as autonomous agents who react to inputs of their own accord. I am not sure how this in any way implies or doesn't imply lack of thought, you will have to explain that mechanism to me since you are so sure about that. They have recorded experience and information in the training corpus, and also a working memory to gather experience, which is smaller of course.
Again what are your experiences? Those are sensory input data fed to a Turing computer (we are Turing computers, like any physical system is).
Human input data is sensory data about the world, which they collect in a way that is directed and attention driven. It involves agency. That sensory data is abstracted, by perception, by memory, and influenced by emotional context and the context of prior memory. Humans then record that in the form of language which is a further abstraction. But each abstraction is a label which is in some chain that can be grounded to subjective experience.
That's not true of an LLM. A LLM is a relation model of tokens (which we recognise as labels to things or concepts), based on harvesting the complex relationships between tokens from a massive corpus of labels - which are taken from (amongst other things) the sum total of human expression of the human experience.
So an LLM can tell me about what it feels like to experience the sun on its skin on the beach, but what it is really doing is regurgitating some sample of human expression about what that is. It can regurgitate a story that lies on the surface of recorded human expression, but it's still a regurgitation. It's just processing tokens.
The map is not the territory.
An LLM can find you a novel new route on the map, but it can't explore new territory, it can just tell you about someone else's map.
In any case, at least I am okay with the acknowledgment of humans and human brains being physical systems. Unfortunately getting even a straight acknowledgement of this fact seems hard for many.
A LLM is not a brain. Brains are the product of selective pressure, LLMs are the product of a scanner, matrix algebra and a loss function.
A brain is successful if it keeps a body alive long enough to successfully breed (not even necessarily its own). To do that it has an almost innumerable collection of competing factors. It's deciding the next action.
An LLM is predicting the next token, often of a string of text that is the product of some brain. If so, it's an entirely conceptual account (words) of some brains' subjective experience that is (at least partially - eg emotion) completely untethered to the world.
These are very, very different things.
For example, why would we expect something whose entire selection pressure is "what happens next" to develop a sense of self?
There is no self, there is only the next token. An LLMs behaviour might make it LOOK like it's doing things that imply it has a self, but that's illusory - it's just imitating the average tokens that humans produce in its current state - because that's the very nature of what it is.
> As long as we have questions to ask, they are not theoretically constrained
> some infinities are bigger than others
Those are both limitations. They need conjectures and they need training data. Can they produce their own conjectures and training data, and feed off it forever, while staying meaningful and not being stuck in a loop (a very large set of repeating cliches) or descending into confused noise?
The end of your response seems to be about fears of “model collapse” from a few years ago, but I am convinced those are strongly oversold. For one thing, there’s no need to gather new training data ever again; the training data for Opus is clearly sufficient for instilling reason, and that’s a sufficient condition for AGI -> RSI -> ASI.