Do you understand how these models work under the hood?
It's literally just picking the next most likely word, based on some fancy math. There is nothing that can create new insights or anything approaching intelligence.
It's literally just picking the next most likely word, based on some fancy math. There is nothing that can create new insights or anything approaching intelligence.
The "fancy math" is a bunch of non-linear functions that encode something derived from series of tokens. I believe we're still exploring what the derived vector spaces "mean" for LLMs, with research inspecting smaller and simplified models to try to make sense of it all.
It's clear that LLMs can "remix" in interesting ways. "ChatGPT, write me a Shakespearean-style sonnet about how LLMs work." (https://pastebin.com/FwzqWJ5W) It's not clear to me that our creativity and intelligence is not the same or a similar process.
Beyond that, I'm not convinced there are definitions of meaning, knowledge, creativity, intelligence, etc, that are both useful and don't refer solely to the output. If I need a solution and none exists yet and something creates it, I will call that creative irrespective of process.
But I don't know anything about anything. :)
How could anyone possibly argue that God and Heaven are fiction? No one can prove they don't exist! I prefer to approach such assertions with skepticism. There is no reason for me to believe the attestation is true, so I may as well believe it to be false.
People remix expressions all the time. Any good training corpus will contain countless examples. If a statistical model can recognize the general semantic patterns that are present in human-remixed expressions, then it should be able to repeat them. So that's a thing that both humans and LLMs can do. Does that mean that LLMs are interesting and useful? Yes! Does it mean we are equivalent? I don't think so.
If I could point out specifically what an LLM can't do, then I would be 99% of the way toward building something that can.
Your argument is missing the whole concept of emergence, which is what is at question here. How does emergence come about, under what circumstances, and what is possible in those circumstance? It's an open question.
Following your logic, one could argue that physics is basic and follows well understood rules. We can do some fancy math to predict what will happen in a given particle interaction. How would these basic particles ever give rise to life and minds with reasoning ability?
Well you and I are an existence proof that basic rules when followed on massive systems can give rise to emergent behavior.
Edit since I can't reply: "I don't like your answer, therefore you aren't engaging in good faith." lol
However I can see you are not engaging in this conversation in good faith so I don't think there is anything else to discus with you.
What if this "categorize and ignore/forget" is causing humans to miss something extremely important in their midst? Bad faith is certainly possible here, but also possible is that this person genuinely believes what they say.
From my perspective, it seems pretty clear that you are the one arguing in bad faith. People are allowed to disagree with you. This person presented a thoughtful critique of your argument. It made plenty of sense to me.
The funny thing 'yes'. Million monkeys cannot do it by randomly typing. They cannot even randomly type at all. Each monkey is intelligent creature, they will follow some patterns. If they did manage to produce Shakspeare they definitely understood something.
Today it's easy to programmatically simulate 1m monkeys typing randomly 10 keys per second. Try it ;)
At nearly every point in the universe, there is a noise: Cosmic Microwave Background Radiation.
What if you listened to the CMBR at a specific point in space, converted that signal into Unicode text, and it just happened to write Romeo and Juliet? Is that intelligence? No. That's selection bias.
So why is ChatGPT able to quote Shakespeare when asked? Because the entire text of the play is in its training corpus, and the infinite permutations of nonsensical CMBR noise is not.
The same with models. On low level there are weights, gradients, matrices. But move up and (wow!) It's nothing like it. It's coherent text or actions. Commands and responses. Put together and you have image recognition and generation from description. It's something new with it's own laws and properties. Just a bit up and we'll get coherent behavior and social creatures. For them we need 'generic' component. It's like soul for life. And LLMs could be it. At least they are the best candidates today.
Hell, try writing a paragraph backwards and tell me that you're not "just picking the next most likely word" as well.
This makes me fear getting old, where this magic machine probably fades, almost certainly leaving me pointing my intention, and starting up sentences, without much happening after. I'm afraid I'll be left with my conscious self, which I don't find as entertaining.
I have never discovered any new insights, but it is apparent that the single most important attribute in discovering a new insight is time spent focused on finding a new insight. Undoubtedly, insight simply comes from executing on even more permutations than regular Joes like me take the time for until the right combination finally lands.
The article uses the concept of irony and the word "because" as examples of fairly sophisticated next-word predictors. You can imagine others that require even more understanding.