I think a missing piece is that ChatGPT doesn't know when it's either confused or out of depth. It starts producing gibberish, which is very hard to separate from the quality answers. Especially if it would just straightforwardly say when it doesn't know the answer, that would be really helpful.

It's easy for ChatGPT to fail the classic turing test because if this. I know that's not the goal here, but for it to pass it does not (or should not?) be able to answer everything but just say when it doesn't know the answer. If it would just say it doesn't know the answer, it would be harder to tell it apart from an human.