The thing that impressed me most about LLMs so far is less the factual correctness or incorrectness of its output but the fact that it appears (!) to understand the instructions that are given. I.e., even if you give it an improbable and outlandish task ("write a poem about kernel debugging in the style of Edgar Allan Poe", "write a script for a Star Trek TNG episode in which Picard only talks in curse words "), it always gives a response which is a valid fulfillment of the task.
Of course it could be that the tasks weren't really as outlandish as they seemed and somewhere in the vast amounts of training data there was already a matching TNG fanfic which just needed some slight adjustments or something.
But those kinds of arguments essentially shift the black box from the model to the training data: Instead of claiming the model has magical powers of intelligence, now the training data magically already contains anything you could possibly ask for. I personally don't find that approach that much more rational that believing in some kind of AI consciousness (or fragments of it).
...but of course it could be. This is why I'd wish for foundation models with more controlled training data, so we can make more certain statements about which responses could be reasonably be pulled from the training data and which would be truly novel.