People go so far as to argue these LLMs aren't just broadly highly intelligent, but sentient... They've been out for a while now and this sentiment seems to be pretty sticky.
It's not such a big deal with ChatGPT because it's so locked down and impersonal. But Bings version has no such restrictions, and spews even more bullshit in a bunch of dangerous ways.
Imagine thinking it's super-intelligent and sentient, and then it starts regurgitating that vaccines are actually autism-causing nanoprobes by the Gates Foundation... Or any other number of conspiracies spread across the web.
That would be a powerful endorsement for people unaware of how it actually works.
I even had it tell me to kill myself with very little prompting, as I was interested in seeing if it had appropriate safeties. Someone not in their right mind might be highly persuaded by that.
I just have this sinking feeling in my stomach that this rush to release these models is all heading in a very, very nasty direction.