1. Fine-tune propaganda into the LLM (e.g. evolution isn't real, or something spicier)
2. Chatbot is so humanlike and convincing, this is very effective.
3. Nobody can inspect the neural net weights to see this, so it flies under the radar.
4. Profit!
The scariest thing to me here is just the idea that someone would want to experiment with teaching their kid with an LLM in the first place, while we don't have any institutions or processes to really stop this (apparently appealing) Idea from happening, and that it is stupidly vulnerable in so many ways.
You can say laissez faire when it comes to rational adults and their use of LLMs, but it's not right to put the risk on kids.