That's a really interesting idea. A LLM that tries to optimize engagement.
* They started optimizing for engagement, which meant making it extra horny for extra money (it sent pics).
* Horny bot forgets consent, refuses "no."
* Lawsuits, bans.
* Horny bot gets censored, main subreddit pins the suicide hotline for a while.
Replika ended up taking the app down and refunding everyone's money. Just kidding! It's still around and they're making a second app focused on "practicing flirting."
Sad to them see it years later being a horny app.
This is like the LLM equivalent of Zawinski's Law (Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.).
As noted on SO [1], the second sentence is important too. People prefer horny LLMs.
[1] https://softwareengineering.stackexchange.com/questions/1502...