[flagged]
Though I had thought you were talking about stuff like eg producing more corn on a given piece of land, or making more furniture from less wood or so. Or even just making better batteries and solar cells.
and what if it's a dead end?
Nothing. Because LLMs can’t spit out anything more than the corpus of information they’ve consumed. LLMs aren’t AGI.
This is analogous to how a calculator cannot output any number that isn't computationally derivable from its programming, yet humans using calculators have discovered new mathematical proofs. The tool augments human capability without being AGI.
Your argument is essentially claiming that because a microscope can't generate new cellular structures, it can't help biologists make new discoveries.
You have a fundamental misunderstanding of how LLMs work, which is why you think they are magical.
Of course if you play the LLM Pachinko machine you can get all sorts of novel output from it, but it’s only useful for certain tasks. It’s great for translation, summarizing (also a kind of translation), and to some degree it can recall from its training corpus an interesting fact. And yes, it can synthesize novel content such as poetry, or adapt an oft-used coding pattern in a flavor specified by a prompt.
What it can’t do is come up with a new idea. At least not in a way better than rolling a dice. It may come up with an idea that you, dear reader, may not have encountered, which makes it great for education.
I don’t have anything more to say, but you’re welcome to continue this discussion with an agent of your choice.