Are they supposed to monitor the content of the chats? Some would call that eavesdropping.
The sad fact is that desperate people exist. And these desperate people are willing to do despicable things to make a buck.
So "Facebook should abort itself" is one option.
AFAIK Onlyfans doesn't directly purse blackmail cases with its own legal teams, it just bans accounts.
FWIW, if the goal is to detect potential scam/abuse scenarios and offer help, then nowadays, they could train a ML classifier and have it run locally. If the conversation gets classified as highly suspect, the app could pop up some "this looks suspect, if you need help press HERE to share the conversation with us" warning. Privacy concerns would go away entirely, if nothing is actually reported to the mothership until the user specifically reports a conversation.
There's only so much they can do for users behind their backs, and a lot of that is undesirable unless you trust the company to be benevolent; local classifiers popping up recommendations could, however, help users help themselves.
"NOTICE: The user appears to be in XXX, Nigeria. The User's hometown and usual IP is XXX , USA. Beware of scams or impersonations by people hijacking your friend's accounts"
But Social media companies would never do this because it would destroy their so called "brand-trust"
Another user noted that OnlyFans is analyzing text and bans anyone saying anything approaching blackmail.
Odd (but maybe not odd at all) to think that OnlyFans might be the leader on this.
But if you can prevent script kiddies, why not? The current barrier to entry is so damn low. Paying VPN is indeed a barrier to entry.
From the point of Facebook, I think they use SMS verification and their app also requires location permission. Facebook is pretty good at preventing bad signups. What they need to do is invest that effort in keeping their user base safe.
You don’t get to say “we will read your messages to sell ads, we will straight up sell your message history to other companies, but no we can’t do anything to protect users from trivially detectable scams.
They used to actually sell your days to select groups.
Now they sell advertiser ads based on your data but the advertiser does not get to see said data.
I rather assumed they already did monitor chat, in order to target adverts, A/B test, handle abuse reports, filter spam, etc.
Even if FB says it has "end to end encryption", I can't fully believe them due to the political pressure under the banner "terrorism/kids get hurt" is immense regardless of us here regarding that as thought terminating cliché from the intelligence agencies.