LLM are just large language models, they're not capable of thinking or coming up with novel ideas. They could enable people to do so by providing food for brainstorming.
The sun is just a bunch of hydrogen.
A computer is just a bunch of transistors.
Humans are just a bunch of cells.
An artificial neural network is just a bunch of matrix multiplications.
I personally think LLMs are extremely limited and overhyped. But this form of argument seems incorrect to me since it can be used to argue that LLMs cannot do things we already know they can do.
LLMs are like mentally challenged autistic human beings. The fact that we even built such a thing is a milestone in humanity.
This claim is unfalsifiable given common definitions of "thinking" and "coming up with novel ideas".
Here's a simple program that does not even do LLMs that will trivially enumerate all ideas (broken UTF8 handling omitted for brevity):
for (var numeral = new BigInteger(0); ; numeral++)
WriteLine(UTF8.GetString(numeral.ToByteArray());