I can definitely picture this being a personal tutor to a child
Not only would I be risking my kids being exposed to things that were outright wrong or entirely inappropriate, but I'd also be missing the opportunity to discuss their questions and the answers with them, provide context, and learn new things myself in the process.
But yes, you're final point is the one I'm getting at: You should be actively monitoring and explaining what your child is being told whether it comes from a human or an LLM.
You don’t need to be a “prompt engineer” to get use out of a language model. You just have to recognize what it is, and what it isn’t.
If that can’t be taught to children, there’s no sense fretting over AI. We don’t need it to fail.
That said, I like the AI, but only as an adult who has enough training to question whether I'm being lied to by an auto regressive Turing machine.
go find examples of the lying tutor you hypothesize - you might just find smart kids extracting value from chaos. your fears will not deny them that.