Comment by OrwellianChild | Hacker News Reader

Comment by OrwellianChild | Hacker News Reader

Parent Full thread

UPDATE: Per @mic47 in a comment below, the video does not actually discuss content moderation on the E2E encrypted systems (Messenger & WhatsApp). It is only discussing the FB website itself, which is already in the open (to Facebook).

Original message follows for continuity:

Thank you for this!

The 4-layers-deep source is an F8 talk on using AI to filter for FB TOS-violating content. The relevant portion of the talk is discussing hyperlink website previews of the kind that show up next to URLs posted in Messenger/WhatsApp that show the title and feature image next to the link. In order for this functionality to work at all, the app must crawl the link to pull the metadata for display. This AI team is attempting to pre-emptively block URL-masking attempts that fool the Preview into showing benign content before redirecting users to bad content.

To do this, FB must "pierce the veil" of E2E encryption in order to see what the URL is and test the content. This opens the possibility that FB can further expose supposedly encrypted content in the chat/message. The actual talk refers to putting this AI on the device itself, preserving user privacy (a difficult technical challenge).

If this leaking of URL data is status quo on FB messaging systems, is it reasonable to assume that the supposed encryption is more or less invalid? Are there any ways that this could be a secure system with this kind of content validation going on?

The presentation actually talks about FB needed to solve challenges of training and evaluating the AI without getting the data to it's server, and I haven't noticed any point where whey would suggest that they need to break encryption, or get the unencrypted data to the servers.

If everything happens on the device and noting is sent to servers, how does it break encryption?

In case I am wrong, and they actually suggest that they will really send data back to its servers, where it is suggested? I really failed to find it.

Upon reviewing the video, I think you are correct. The video is discussing content filtering through AI, and whether it can be done on the device as opposed to the server, but the images accompanying the talk show a normal FB post on the website - not a Messenger window. It is not implied or suggested that they are screening content on their E2E encrypted platforms (Messenger and WhatsApp). What the speaker says:

...So far, we have been keeping this fight [against bad actors and harmful content] on familiar grounds. And that is, we have been training our AI models on the server and making inferences on the server when all the data are flooding into our data centers.

While this works for most scenarios, it is not the ideal setup for some unique integrity challenges. URL masking is one such problem which is very hard to do. We have the traditional way of server-side inference. What is URL masking? Let us imagine that a user sees a link on the app and decides to click on it. When they click on it, Facebook actually logs the URL to crawl it at a later date. But... the publisher can dynamically change the content of the webpage to make it look more legitimate [to Facebook]. But then our users click on the same link, they see something completely different - oftentimes it is disturbing; oftentimes it violates our policy standards. Of course, this creates a bad experience for our community that we would like to avoid. This and similar integrity problems are best solved with AI on the device.

Yeah but that URL thing is entirely FB's own choice and own problem. Definitely way more a thing of "violates our policy standards" than "oftentimes it is disturbing" because people going around tricking others to click on shock videos using advanced cloaking techniques ... just isn't that big of a problem.

Linking to things is a normal thing to do on the Internet, but Facebook wants to control that, because links often also mean the user moves their attention away from FB, onto the linked thing. This is what FB doesn't want. But it is a very useful thing for the user, to be able to smoothly navigate between apps and websites depending on their needs.

I do wonder why they chose to go with the danger of URL masking that links can be "disturbing" instead of "scams". Because I don't believe these techniques are (widely) being used to trick people into watching animal torture or other disturbing things. They'll be tricked to click on scams and phishing sites.

The difference between "disturbing" and "scam" is literally the difference between subjectively not-okay and objectively not-okay.

Facebook just wants the power to prohibit you from accessing what it deems "disturbing".

Also just from a technical perspective, there's some very weird assumptions: FB's server would see something different than the user's browser. But FB's local AI spy process would see the same thing? Are they seriously pretending that you shouldn't be able to securely log onto another website in a browser? Yes the user's browser sees something different, because it is logged in, and no FB's local AI spy process is not invited ...

At most they could warn that the site looks different than what the server side FB scanner saw. Then the user could decide if they feel more secure because this is expected as they are logged in and FB can't watch along, or if this is unexpected and they should be cautious.

> If everything happens on the device and noting is sent to servers, how does it break encryption?

if the process runs on the device, scans information on the device, and then triggers to notify FB when it matches something, that's breaking the encryption.

it's leaking of encrypted information (the fact that it's there, which rules triggered), but more importantly what's the use of encryption if you got a process that's basically acting as a mole, even if it's running locally?

unless I am wrong how this works, but at some point it'll trigger and notify FB or perhaps a government? if it doesn't do anything, then it might as well no be there.

and if this system gets off, there will be false positives, and there will be trigger rules matching much broader things than they strictly need, "just in case". we kind of know how these stories go by now and which bits we can safely give no benefit of the doubt because a state or a corporation wants them so strongly. I'm gonna guess it will be a black box "algorithm" without clear trigger rules, because it's AI/ML it's hard/impossible to explain exactly how it reaches its conclusions and therefore nobody knows really (and much NDA unto those who do). that's gonna be a hard sell though, even to the general public.

Reply on news.ycombinator.com