Hacker News Reader

Comment by mrshadowgoose | Hacker News Reader

Is this just a smokescreen around slowly sneaking CSAM scanning back in after the pushback last time? The "default on" behavior is suspect.

[1] https://www.wired.com/story/apple-photo-scanning-csam-commun...

View on HN

My thoughts exactly: "we've got this crafty image fingerprinting, the CSAM detection use proved too controversial to roll out, but let's get the core flows into something that sounds useful for users, so the code atays alive, improving, & ready for future expansion."

Whether such fingerprinting can reliably be limited to public "landmarks" is an interesting question, dependent on unclear implementation details.

Even if the user-visible search is limited to 'landmarks', does the process pre-create (even if only on-device) fingerprints of many other things as well? If so, it suddenly becomes possible for briefly-active non-persistent malware to instantly find images of interest without the wider access & additional processing it'd otherwise take.

> let's get the core flows into something that sounds useful for users

is it even that?

I don't see the benefit of this whatsoever

The search feature is useful at times, and while local processing is good enough to find (some of the) photos I've taken that match a search term like "table", it can't currently find a photo from a search term of "specific neighbourhood in my city" or "name of specific mountain I climbed years ago" - so if by processing on their servers allows them to do that then it would be genuinely beneficial.

But not beneficial enough to make up for the loss of privacy, so I've disabled it without finding out how useful or not the functionality is.

Yup, this is their way of injecting the "phone home" element via an innocuous rationale, "location matching". The global index will of course also match against other markers they deem worthy of matching, even if they don't return that to the user.

But wouldn't the homomorphic encryption prevent Apple's servers from knowing if there was a match or not?

The server must know what it's matching at some point, to be able to generate a response:

> The server identifies the relevant shard based on the index in the client query and uses HE to compute the embedding similarity in this encrypted space. The encrypted scores and set of corresponding metadata (such as landmark names) for candidate landmarks are then returned to the client.

Even with the server supposedly not knowing the identity of the client, the response could simply include extra metadata like some flag that then triggers an instant send of that photo to Apple's (or law enforcement's) servers unencrypted. Who knows?

[0] https://machinelearning.apple.com/research/homomorphic-encry..., during the period of generating

> The server must know what it's matching at some point, to be able to generate a response

The entire point of homomorphic encryption is that it doesn't.

The homomorphic encrypted Wikipedia lookup example is pretty neat.

https://spiralwiki.com/

https://news.ycombinator.com/item?id=31668814

Oh, if that's the case, they really could have explained that better. The language used in Apple's article doesn't explain that the server cannot know the query or result (it implies as such, but doesn't make this clear, nor explain how/why)

> they really could have explained that better. The language used in Apple's article

This one?

https://machinelearning.apple.com/research/homomorphic-encry...

I find that description perfectly clear for someone who doesn't already know what homomorphic encryption means:

> One of the key technologies we use to do this is homomorphic encryption (HE), a form of cryptography that enables computation on encrypted data (see Figure 1). HE is designed so that a client device encrypts a query before sending it to a server, and the server operates on the encrypted query and generates an encrypted response, which the client then decrypts. The server does not decrypt the original request or even have access to the decryption key, so HE is designed to keep the client query private throughout the process.

Later:

> HE excels in settings where a client needs to look up information on a server while keeping the lookup computation encrypted.

And there's more perfectly layman-understandable in PIR and PNSS sections, complete with real-life examples and simple diagrams.

One just has to read the goddamn thing, which apparently is an insurmountably tall order these days for content that is more than 250 characters.

I read that. It doesn't actually explain how the server can tell me the Eiffel Tower is in my photo without knowing it's telling me that. It glosses over the mechanism by which it can tell me what's in my photo without the service itself knowing. Yeah, cool, never-decrypted. So how do they match up? An ultra-abstract "A[encrypted] + B[encrypted] = A+B[encrypted]" doesn't tell me anything.

As an aside, your unfounded derision about "RTFA" is unwarranted and unnecessary. That was an unusually-hostile response for like, just talking about things.

The server doesn't tell you (and doesn't know) that it's the eiffel tower. Only when the result comes back to your device, is your device able to identify it as such.

I'll take it as if you don't know what homomorphic encryption means.

caveat: this is not rigorous math, just a rough presentation for one to wrap their head around the keystone principles of it

Say you have a space like natural numbers N and a computation function f: N->R. I'm just using R - as in result - so that it's not confusing below:

Take a n in N, f(n) = r, r in R

That's nice, we compute something, but the "computation place" is going to know both n and r.

Now imagine you have an encryption function g that maps every n in N to another integer. I'll just mark encrypted things with ' e.g the encrypted space is N' for easier trackability, so g: N->N' is encryption and its inverse g': N'->N is decryption.

So take a n in N, g(n) = n' in N', and g'(n') = n back again.

    N            N'
    
    n --- g ---> n'
    
    n <-- g' --- n'

That's basic encryption.

Now back to making f useful: the idea is to have an encryption scheme that allows performing a certain set of operations on the encrypted space and get an encrypted result that results in an identical result once decrypted. From these operations we can have a f' function so that:

for all n in N, f(n) = g'(f'(g(n)))

Visually:

    N           N'            R'            R
    n --- f --> n' --- g' --> r' --- f' --> r

The whole point is that cheap f and f' happen locally and intensive g' can happen in untrusted places, because all that is seen is an encrypted input n' and encrypted result r'. Only thing it can see is at best that there's A match but not what THE match is, because that all just looks like random numbers; even the no-match case could very well be encrypted and in that case it doesn't even see that.

That is the principle of "homomorphic encryption", and it carries the intrinsic property of the computation place not knowing a single thing about neither the input or the output.

> It doesn't actually explain how the server can tell me the Eiffel Tower is in my photo without knowing it's telling me that.

So it depends on what you mean by "explain how":

- "homomorphic encryption" is the high level answer, described and provided in the Apple article

- if that is not a sufficiently detailed answer then you gotta roll your sleeves up and dive much deeper: the swift-homomorphic-encryption implementation, the BFV paper, the Wally paper, and the other bits that help prevent accidental side channel leaking are all both superficially discussed in a synthetic way and deep-linked in the Apple article in "would you like to know more" fashion.

Or maybe you wanted to know how they performed the actual match (g') in the encrypted space?

> With PNNS, the client encrypts a vector embedding and sends the resulting ciphertext as a query to the server. The server performs HE computation to conduct a nearest neighbor search and sends the resulting encrypted values back to the requesting device, which decrypts to learn the nearest neighbor to its query embedding

Which is possible because they

> have implemented the Brakerski-Fan-Vercauteren (BFV) HE scheme, which supports homomorphic operations that are well suited for computation (such as dot products or cosine similarity) on embedding vectors that are common to ML workflows.

My apologies for the harsh words, they were not meant as an ad hominem, more like a general ranty reflection on the state of the web society.

The setup for “that wasn’t real homomorphic encryption!” is in, when in 2-4 years it comes out that they were doing this exact thing.

The entire concept of a homomorphic encryption system is a land mine outside of obscure academic discussions. In practice systems marketed to the public as “homomorphic encryption” will result in user data exfil mark my words.

(btw thanks for the links!) :)

not if you need to access from multiple devices (otherwise, what's the point of this feature?)

in that case it's the source of common key of "the same account" becomes the threat

and now you have to trust... megacorporation with closed-garden ecosystem... to not access its own servers in your place?

>not if you need to access from multiple devices (otherwise, what's the point of this feature?)

I don't think the feature works perfectly fine on single device. You take a ton of pictures on your iPhone. You search your photos for "Eiffel tower" and it shows you the photos you took of the Eiffel tower. I don't see why you need multiple devices.

Honestly, why the hell would Apple bother with such a contrived and machiavellian strategy to spy on their users?

They literally own the code to iOS. If they wanted to covertly track their customers, they could just have their devices phone home with whatever data they wanted to collect. Realistically there would be no way to know if this was actually happening, because modern devices emit so much encrypted data anyway, it wouldn’t be hard to hide some nefarious in all the noise.

Time Cook isn’t some Bond villain, sitting in a giant chair, stroking a white cat, plotting to take over the world by lulling everyone into a false sense of privacy (I mean Zuckerburg already did that). Apple is just a large giant corporation that wants to make money, and is pretty damn open about that fact. They clearly think that they can make more money by doubling down on more privacy, but that doesn’t work if you don’t actually provide the privacy, because ultimately, people are really crap at keeping secrets, especially when a media group would happily pay for a story, even at Apple.

Yeah, that sorta already exists. If you've ever done remote customer support, they can send a query to remotely-view your screen -- a query which you have to accept or deny. There's really zero reason there couldn't be a similar feature, but without asking you, and without putting a big red bar a the top of your screen that says "Apple Support is viewing your screen". Wish I had a screenshot or photo of that, can't seem to find a screenshot online unfortunately.

Exactly like how Microsoft "backed off" Recall. Uuuuuntil they shoved it back in and made it undeleteable.

By removing it from the market, making enormous technical tweaks based on user feedback, and then putting it back on the market.

Yes my thoughts as well. The tech was so expensive I guess that they had a need to test / run it to proof it’s private? I mean the model to find landmarks in your photos could run locally as well or? Ok I’m not 100% sure here.

I assume that the model couldn’t run locally for some reason. Probably either uses too much power or needs too much memory.

No, it is not. Whatever their other failings, Apple doesn’t think that way.

The cynical reason: consider that you can’t plan engineering features of this scale without written documentation, which will always surface in court.

The prima facie reason: Apple genuinely wants to provide useful features requiring server participation.

This is incredibly naive and curiously defensive.

If this was a feature on its own then it would not be popular.

Citing national security, some danger will justify its existence.

Apple alone does not control and dictate what goes in, once you reach their level of size and wealth that exceed even developed countries, you ultimately cannot be the controller of your destiny purely as a profit orientated corporation.

ex) Meta, Microsoft, Google

Very likely yes. Why else would they add a feature that incurs costs for them as an update, at no cost to the users (and not even make a fuss about it)?

It is obvious they are monetizing this feature somehow. Could be as innocuous as them training their AI dataset, or feeding into their growing ad business (locations and other things identified in the photos), or collaboration with law enforcement for various purposes (such as notifying the CCP about people's Winnie-the-Pooh memes), or a lot more ominous things.

> Very likely yes. Why else would they add a feature that incurs costs for them as an update, at no cost to the users (and not even make a fuss about it)?

Erm, you’re aware of the whole Apple intelligence thing right? An entire product that costs Apple money, provided at “no cost” to the user (if you had an iPhone 15). Also every feature in an OS update has a costs associated with it, and iOS updates have cost money for the best part of a decade now.

Has it occurred to you that reason Apple includes new features in their updates is to provide customers with more reasons to buy more iPhones? Just because feature are provided at “no cost” at point of consumption, doesn’t mean Apple won’t make money in the long run, and selling user data isn’t the only way to monetise these features. Companies have been giving out “freebies” for centuries before the internet existed, and the possibility of large scale data collection and trading was even imaginable.

Of course, but those would be features in the product code deployed on users' devices (one time investment), not a service that has ongoing costs of operation associated with each call. They wouldn't just give this out for free (especially to older iPhones), it makes no business sense. If you're not paying for a product, you are the product!

That whole incident was so misinformed.

CSAM scanning takes place on the cloud with all the major players. It only has hashes for the worst of the worst stuff out there.

What Apple (and others do) is allow the file to be scanned unencrypted on the server.

What the feature Apple wanted to add was scan the files on the device and flag anything that gets a match.

That file in question would be able to be decrypted on the server and checked by a human. For everything else it was encrypted in a way it cannot be looked at.

If you had icloud disabled it could do nothing.

The intent was to protect data, children and reduce the amount of processing done on the server end to analyse everything.

Everyone lost their mind yet it was clearly laid out in the papers Apple released on it.

That technology of perceptional hashes could have failed in numerous ways, ruining lives of law-abiding users along the way.

The chance of a hash colliding is near 0%. The hashes are for some of the worst content out there, its not trying to detect anything else.

Even so a human is in the loop to review what got a hit. Which is exactly currently happens now.

> The chance of a hash colliding is near 0%

Until someone finds a successful collision attack.

> Even so a human is in the loop to review what got a hit.

Until shareholder/growth pressure causes them to replace that human with an AI.

> Until someone finds a successful collision attack.

Indeed, and within hours of the hash function being made available to me I developed a second preimage attack (strictly stronger than a collision attack)... allowing me to modify images in a visually non-objectionable way to match an arbitrary hash value.

> Until shareholder/growth pressure causes them to replace that human with an AI.

Indeed, but more than that:

The "human" is still a dire privacy loss. Perhaps Apple's review might have protected you from some erroneous reports to law enforcement, but does it protect you from an apple-employed stalker ex? does it protect you from paparazzi? Does it protect you from the double thinking ("do I photograph my kids playing in the sprinklers? do I take a nude photo of my spouse?") due knowing that your private activity is being watched?

One could easily argue that some AI second level review is an "improvement", which is another reason why your assumption that even that backstop would eventually be removed is a reasonable one.

> Until someone finds a successful collision attack.

The only way to get a successful collision attack is to have a picture/video that is stored in that hash database.

To put it in perspective. You randomly picking the same atom twice in the observable universe has a higher chance than getting a mismatch.

> Until shareholder/growth pressure causes them to replace that human with an AI.

How do you think it will impact shareholders to know that the company is not actively scanning for illegal content.

Also it's Interpol/FBI that get involved.

> The chance of a hash colliding is near 0%.

The 'chance' is 100% -- collisions and even arbitrary second preimages have been constructed.

> The hashes are for some of the worst content out there, its not trying to detect anything else.

You don't know that because apple developed powerful new cryptographic techniques to protect themselves and their data providers from accountability.

> collisions and even arbitrary second preimages have been constructed.

The chance of a mismatch is 8.63616855509e-78%

If the hash was an atom then you would have to guess which atom in the observable universe it is. That is how likely a collision will happen.

I have posted numerous neuralhash collisions online already[1] and can generate more on demand. The "chance" is 100% because perceptual hashing schemes do not and cannot achieve cryptographic strength.

[1] https://academic.oup.com/cybersecurity/article/10/1/tyad020/... see the girl/dog image on page 12 for one of my examples in the academic literature.

Apple sells their products in oppressive regimes which force them to implement region specific features. E.g. China has their own iCloud, presumeably so it can be easily snooped on.

If they were to add this anti-CSAM feature, it is not unreasonable to think that Apple would be forced to add non-CSAM stuff to the database in these countries, e.g. anything against a local dictatorship/ etc. Adding the feature would only catch the low hanging CSAM fruit, at the cost of great privacy and probably human life. If it was going to stop CSAM once and for all, it could possibly be justified, but that's not the case.

If China can force Apple to do that stuff, then it can do that regardless of whether or not they add this feature.

Apple and others already scan peoples pictures/videos for this stuff, so your argument can be applied to what it is now.

Apples suggestion would have meant your data would be more protected as even they would not have been able to unencrypt your data.

Yes this is better than upload the entire photo. Just like virus scan can be done entirely on device, can flagging be local?. If homeomorphic encryption allows similarity matching, does not seem entirely private. Can people be matched?

"It only has hashes for the worst of the worst stuff out there." [citation needed]

I know someone whose MS account was permabanned because they had photos of their own kid in the bathtub. I mean, I guess the person could have been lying, but I doubt they would even have been talking about it if the truth was less innocuous.

Sure, and they do that because Microsoft's CSAM detection product (which other providers like Google supposedly use) operates by having unencrypted data access to your files in the cloud.

What Apple wanted to do is do those operations using homomorphic encryption and threshold key release so that the data was checked while still encrypted, and only after having a certain number of high likelihood matches would the possibility exist to see the encrypted data.

So the optimistic perspective was that it was a solid win against the current state of the industry (cloud accounts storing information unencrypted so that CSAM products can analyze data), while the pessimistic perspective was that your phone was now acting as a snitch on your behavior (slippery slope etc.)

I'm just refuting what the person I responded to said, because apparently these services have hashes for more than just "the worst of the worst stuff" or whatever.

> because apparently these services have hashes for more than just "the worst of the worst stuff" or whatever.

Do you have a citation for that as well? I linked you what CSAM is. Where are you getting your information from?

> while the pessimistic perspective was that your phone was now acting as a snitch on your behavior

The actual auditing doesn't happen until the file hits the cloud though. Which is what happens now.

Thanks for some voice of reason. I'm still amazed at how many are still upset about this but clearly never actually read the paper on it.

> [citation needed]

It is actually detailed in Apples paper. Also:

https://www.interpol.int/en/Crimes/Crimes-against-children/I...

It works by generating a hash on known materials. Those hashes are shared with other companies so they can find that material without having to see the horrific stuff. The chance of a hash collision was also detailed in the paper which is so low to be non-existent. Even if a clash occurs a human still reviews the materials, and it normally needs a couple of hits to trigger an audit (again according to apples paper on it).

> I know someone whose MS account was permabanned because they had photos of their own kid in the bathtub

So you ask me for a citation and then give me anecdotal evidence?

Even if that happened it has nothing to do with CSAM.

> Everyone lost their mind yet it was clearly laid out in the papers Apple released on it.

And people working with CSAM and databases of CSAM have said it was a very bad idea.

Citation needed. As the latest news suggests the opposite.

> Citation needed.

The best one I remember is this one: https://www.hackerfactor.com/blog/index.php?/archives/929-On...

> As the latest news suggests the opposite.

What news?

They don't work on the CSAM. They are a cloud provider who makes CSAM reports.

The only thing I see that is wrong in what they claimed is this:

> The problem is that you don't know which pictures will be sent to Apple.

Apple said in their paper that they don't send anything that is flagged. When the cloud sync occurs the files that are flagged get reviewed. All other files remain encrypted.

> What news?

https://www.nytimes.com/2024/12/08/technology/apple-child-se...

> They don't work on the CSAM. They are a cloud provider who makes CSAM reports.

--- start quote ---

If you sort NCMEC's list of reporting providers by the number of submissions in 2020, then I come in at #40 out of 168. For 2019, I'm #31 out of 148

I repeatedly begged NCMEC for a hash set so I could try to automate detection. Eventually (about a year later) they provided me with about 20,000 MD5 hashes that match known CP. In addition, I had about 3 million SHA1 and MD5 hashes from other law enforcement sources.

--- end quote ---

Somehow this isn't "working with CSAM or CSAM databases", oh well.

> Apple said in their paper that they don't send anything that is flagged.

This is literally addressed in the article you pretend you read.

> https://www.nytimes.com/2024/12/08/technology/apple-child-se...

I don't have access to this article. How does it show that people working with CSAM say Apple's implementation is a good idea?

> The intent was to protect data, children and reduce the amount of processing done on the server end to analyse everything.

If it’s for the children, then giving up our civil liberties is a small price to pay. I’d also like to give up liberties in the name of “terrorism”.

When we willingly give up our rights out of fear, these evil people have won.

> If it’s for the children, then giving up our civil liberties is a small price to pay.

All your pictures and videos are currently scanned. What civil liberty did their approach change in that?

I can't believe how uninformed, angry, and still willing to argue about it people were over this. The whole point was a very reasonable compromise between a legal requirement to scan photos and keeping photos end-to-end encrypted for the user. You can say the scanning requirement is wrong, there's plenty of arguments for that. But Apple went so above and beyond to try to keep photo content private and provide E2E encryption while still trying to follow the spirit of the law. No other big tech company even bothers, and somehow Apple is the outrage target.

There isn’t a law that requires them to proactively scan photos. That is why they could turn the feature back off.

A law by the government requiring proactive scanning of photos would in fact make the whole situation worse in the US because there would need to be a warrant if the government is requiring the scan. As long as it's voluntary by the company and not coerced by the government, they can proactively scan.

> a legal requirement to scan photos

Can you provide documentation demonstrating this requirement in the United States? It is widely understood that no such requirement exists.

There's no need to compromise with any requirement, this was entirely voluntary on Apple's part. That's why people were upset.

> I can't believe how uninformed

Oh the irony.

Should have said "potential legal requirement". There was a persistent threat of blocking the use of E2E encryption for this exact reason.

> Can you provide documentation demonstrating this requirement in the United States?

PROTECT Act of 2003 - Details CSAM materials as being illegal which is enforced by FBI + ICE.

Also NCMEC which is a non-profit created by the US government that actively works in this area.

> a legal requirement to scan photos

There is absolutely no such legal requirement. If there were one it would constitute an unlawful search.

The reason the provider scanning is lawful at all is because the provider has inspected material voluntarily handed over to them, and through their own lawful access to the customer material has independently and without the direction of the government discovered what they believe to be unlawful material.

The cryptographic functionality in Apple's system was not there to protect the user's prviacy, the cryptographic function instead protected apple and their datasources from accountability by concealing the fingerprints that would cause user's private data to be exposed.

> What the feature Apple wanted to add was scan the files on the device and flag anything that gets a match.

This is not the revelation you think it is. Critics understood this perfectly.

People simply did not want their devices scanning their content against some opaque uninspectable government-controlled list that might send you to jail in the case of a match.

More generally, people usually want their devices working for their personal interests only, and not some opaque government purpose.

From my understanding, it didn't scan all of the files on the device, just the files that were getting uploaded to Apple's iCloud. It was set up to scan the photos on the device because the files were encrypted before they were sent to the cloud and Apple couldn't access the contents but still wanted to try to make sure that their cloud wasn't storing anything that matched various hashes for bad content.

If you never uploaded those files to the cloud, the scanning wouldn't catch any files that are only local.

Your understanding is correct, as was/is the understanding of people critical of the feature.

People simply don't want their device's default state to be "silently working against you, unless you are hyperaware of everything that needs to be disabled". Attacks on this desire were felt particularly strongly due to Apple having no legal requirement to implement that functionality.

One also can't make the moral argument that the "bad content" list only included CSAM material, as that list was deliberately made opaque. It was a "just trust me bro" situation.

> People simply don't want their device's default state to be "silently working against you

That was the misconception of what was happening though.

Nothing happens on your device. Only when it gets to the cloud. It just puts a flag on the picture in question to have the cloud scan it.

Which is exactly what happens before Apple suggested it and happens now. Except it does it for all your files.

> One also can't make the moral argument that the "bad content" list only included CSAM material, as that list was deliberately made opaque. It was a "just trust me bro" situation.

CSAM database is run by Interpol. What evidence do you have that they are not being honest?

The scanning and matching is performed on your own device, against a copy of the databases which is encrypted to protect apple and their data providers against accountability for its content. The result of that match is itself encrypted, owing to the fact that the database is encrypted. On upload the query is decrypted and if there are above a threshold matches the decryption keys to all your content are revealed to apple.

Your phone is your most trusted agent-- it's a mandatory part of your life that mediates your interactions with friends, family, lovers, the government, your doctors, your lawyers, and your priest. You share with it secrets you would tell no other person. It's always with you, tracking your location and recording your activities. And in many cases its use is practically mandated. I think it's inappropriate for such a device to serve any interest except your own.

While it is true that the original proposal operated only on images that you would upload to icloud many people assumed the functionality would be applied more widely over time. This article seems to have proved that point: Apple is now applying essentially the same scanning technology (this time they claim the databases is of "landmarks") to otherwise entirely local photos.

> Your phone is your most trusted agent

What makes you think your phone is the most trusted device? Do you know everything that is running on your phone and what it does?

If a government wanted to enforce it then none of what Apple suggested mattered.

In fact they wouldn't even need to release a paper explaining what they planned to do.

This has been the fallacy against the whole argument.

My message was an informal argument. Apple has proposed and (now) applied the same spyware technology to their desktop/laptop operating system as well. But for most people in the US their phone absolute does occupy that most-trusted niche. For better or worse. The fact that this trust may currently be ill-advised is all the more reason people should demand change that makes it possible.

> If a government wanted to enforce it then none of what Apple suggested mattered.

Perhaps you live in a dicatorship. If so, I'm sorry. In the united states the power of the government is limited by the constitution. The kind of automated surveillance performed nominally 'consensually' via corporations would be unambiguously unlawful for the government to perform.

> People simply did not want their devices scanning their content against some opaque uninspectable government-controlled list that might send you to jail in the case of a match.

Again I feel like many people just didn't read/understand the paper.

As it stands now all your files/videos are scanned on all major Cloud companies.

Even if you get a hit on the database the hash doesn't put you in jail. The illegal materials do and a human reviews that before making a case.

Reply on news.ycombinator.com