This is already a real issue with many existing public chatbots.
Take r/replika for example, they’re constantly getting posts from users who think their chatbot partner is real. It’s the first question in their FAQ.
The mods keep needing to post reminders. [0] Even in this post, there are many commenters (if you scroll down) who don’t believe the mods.
[0] https://old.reddit.com/r/replika/comments/r5t0wk/so_you_thin...