They are 'next word prediction models' which elicit some kinds of reasoning embedded in our language, but it's a crude approximation at best.
The AGI metaphors are Ayahuasca Koolaid, like a magician duped by his own magic trick.
There will be no AGI, especially because there will be not 'automaton' aka distinct entity that elicits those behaviours.
Imagine if someone proposed 'Siri' were 'conscious' - well nobody would say that, because we know it's just a voice-based interface onto other things.
Well, Siri is about to appear much smarter thanks to LLMs, and be able to 'pass the bar exam' - but ultimately nothing has fundamentally changed.
Whereas each automaton in the human world had it's own distinct 'context' - the AI world will not have that at all. Context will be as fleeting as memory in RAM, and it will be across various systems that we use daily.
It's just tech, that's it.