Cool, so clearly articulate the goal posts. What do LLMs have to do to convince you that they are intelligent? If the answer is there is no amount of evidence that can change your mind, then you're not arguing in good faith.
Should we consider it our equal or superior to us ? Should we give it the reigns of politics if it’s superior in decision making ? Or maybe the premise is « given all the knowledge that exists coupled with a good algorithm, you look/are/have intelligence » ? In which case intelligence is worthless in a way. It’s just a characteristic, not a quality. Which makes AI fantastic tools and never our equal ?
Come on. If you are actually entertaining the idea that LLMs can possibly be intelligent, you don't know how they work.
But to take your silly question seriously for a minute, maybe I might consider LLMs to be capable of intelligence if they were able to learn, if they were able to solve problems that they weren't explicitly trained for. For example, have an LLM read a bunch of books about the strategy of Go, then actually apply that knowledge to beat an experienced Go player who was deliberately playing unconventional, poor strategies like opening in the center. Since pretty much nobody opens their Go game in the center (the corners are far superior), the LLM's training data is NOT going to have a lot of Go openings where one player plays mostly in the center. At which point you'll see that the LLM isn't actually intelligent, because an intelligent being would have understood the concepts in the book that you should mostly play in the corners at first in order to build territory with the smallest number of moves. But when faced with unconventional moves that aren't found anywhere on the Internet, the LLM would just crash and burn.
That would be a good test of intelligence. Learning by reading books, and then being able to apply that knowledge to new situations where you can't just regurgitate the training material.
This is actually a good illustration of my point that LLMs, as they currently exist, aren't capable of general intelligence. The LLM can give the illusion of learning, because it can go back within its context window and look at the information presented there. But start a new context window in a new browser tab, and the information from the other browser tab isn't there. It's gone.
The ability to learn requires being able to retain the learned concepts for longer than a single conversation. LLMs, by their nature, aren't capable of that. If AGI is going to be achieved, it will need to happen via a different technology than large language models.
EDIT: I should add that the LLM only invented the new language via careful prompting by the user; it didn't do any of the creative-thought work itself, only responding to the user prompting it "Now generate ten more words that match these syntax rules".