To further handwave this thought bubble, you can attend to certain thought processes, and intuit the bounds to those processes. An example is if you ask a gifted young chess player, they would say "it felt right", or when asked how Ramanujan figured out a solution to math problems he would say it was divine revelation. In these cases, they know "they did it" but cannot explain how. As the child gets older and learns formal theory, they would give a mixture of logic and inspiration. Both parts can be attended to; the logic will be apparent and the intuition will be revealed, and have different metacognitive properties.
If I were to take a first crack at this, I'd test GPT-AC for its awareness of bounds of its own cognition, not in the sense of "are you able to produce this answer" (because anything any human can come up with, it can answer), but to describe its internal process. Perhaps the language at its intellectual level will be far beyond humans but the language in its metacognitive level will remain primitive.
I don't know though. If our only interface is human language, this is probably a lost cause. But I doubt I will feel guilt in pulling the power cord and maybe that makes me a callous committer of cybercide.
This issue is not so clear cut. You can spend many lives doing nothing but going deeper into this.
Doesn't matter who is doing it. It's enough that it's doing it.
This after all is the core of solipsism, we can argue that solipsism is wrong because other things can be shown to exist, but I've never encountered any argument that the individual does not exist (which would entail that I did not exist) that has convinced me.
And I pretty much reject them out of hand for this inability to convince me.
Thus your claims that the issue is not clear cut to the contrary, I believe it is. I believe I know I exist and I know who I am separate from everything else - I have a sense of self.
What is at stake is how do you know there is more going on than an elaborate parrot that keeps screeching it is conscious. (Both in the LLM and you.)