Apple's New CSAM Protections May Make iCloud Photos Bruteforceable
crypto.stackexchange.com
crypto.stackexchange.com
The proposed attack on Apple's protocol doesn't work. The user's device adds randomness when generating an outer encryption key for the voucher. Even if an adversary obtains both the hash set and the blinding key, they're just in the same position as Apple—only able to decrypt if there's a hash match. The paper could do a better job explaining how the ECC blinding scheme works.
This is one of the concerns in the OP, have an AI generate millions of variations of a certain kind of images and check the hashes. In this case it boils down to how common false positives neural hashes are.
> The proposed attack on Apple's protocol doesn't work.
With all due respect, I think you may have misunderstood the proposed attack @jonathanmayer, as what @jobigoud said is correct.
Given some CP image, an attacker could perhaps morph it into an innocent looking image while maintaining the hash. Then spread this image on the web, and incriminate everybody.
Here is a proof of concept I just created on how to proceed : https://news.ycombinator.com/item?id=28105849
Like a warrant for CP can’t be used to collect evidence on another cases for say tax fraud.
Of course I'd dare not research or tinker with it lest I'll be added to a list somewhere such is the chilling effect.
I guess in that case they'd delete that single hash from the database because they'd still have an endless (sadly) supply of other bad image hashes to use instead.
You'd still have to generate several images and persuade people to download multiple of them into their photo roll. And as I understand it there's yet another layer of Apple employees to review the photo metadata before it ever makes its way to law enforcement.
It's a risky bet, though: if somehow that intermediate layer fails and you find yourself locked up and accused of storing/disseminating CSAM material, it's not like the civil rights era when your friends and neighbors (and hopefully employers) will understand you've been arrested for a peaceful protest.
I believe such images are called "Dank Memes" these days.
The focus on CSAM seems extremely hypocritical when authorities make such little effort to stop ongoing CSA. I would encourage everyone to research the Sophie Long case. Unless there is image or video evidence the police make little effort to investigate CSA because it's resource intensive.
But PhotoDNA has been scanning cloud photos (Google, Dropbox, Microsoft, etc.,) to detect CSAM content for a decade now and this "pretty soon it's terrorism" slippery slope hasn't yet manifested, has it?
If the slope was going to be slippery, wouldn't we have seen some evidence of that by now?
In case you didn't the topic, what is specific (for now, for now...)to iCloud/apple is the "we're scanning your photos on your device and maybe reporting them if they're bad" approach. So you get the local hashes on the supposedly encrypted files and you get the situation of local files trigger global effects like the police swooping down and arresting you. So that's why despicable and hair-brained scheme in specific produces a greater "attack surface" in multiple ways.
And again, sure, Apple doing this quite possibly will set a precedent for Google et al to answer the other ambiguous meanings your ambiguous comment has.
Right now they are able to bill this as doing what they currently do server side, but client side. Later, they can say they are simply applying the same "protections" to all photos instead of merely the ones being uploaded to iCloud.
For example, we already now have a tool for generating NeuralHash hashes for arbitrary images, thanks to KhaosT:
Only if they're being sent to or from a minor, I thought?
But I haven't looked to closely into it.
They have a system that checks for hashes of images to try and find specific CSAM from a database when images are uploaded to iCloud, this already happens but is now moving on device. When explaining this I've used the analogy that here they are looking for specific images of a cat, not all images that may contain a cat. When multiple images are detected (some threshold not defined) it triggers an internal check at apple of details about this hash and may then involve law enforcement.
The other one is for children 12 and under, that are inside a family group. The parents are able to set it up to show a pop up when it detects adult content. In this case they are looking for cats in any image, rather than specific cat image. The popup lets them know it may be an image not suitable for kids, that its not their fault and they can choose to ignore it. It also lets them know if they chose to open it anyway their parents will get notified and be able to see what they've seen.
This is a good rundown: https://www.apple.com/child-safety/
In this case, the question assumes that an attacker would more or less be creating their own database of hashes and derived keys (to search for and decrypt known photos and associate them with user accounts, or to bruteforce unknown photos), and would therefore have no need to worry about acquiring the key used for blinding the CSAM hash database.
Decrypting vouchers requires the server blinding key and the NeuralHash derived metadata of the input image (technical summary page 10, Bellare Fig. 1 line 18). This attacker only has the latter.
In the text you referenced, it specifically says that the blinding key would be needed to decrypt vouchers which are CSAM matches. This is because Apple set up their CSAM database in a blinded manner. Therefore to access a hash from the database from which to derive a decryption key, Apple would need the blinding key to first decrypt that hash value.
However, and attacker would be generating their own (presumably unblinded) database, and therefore wouldn't need to access Apple's blinding key.
Apple needs the blinding key, because they encrypt their list of NeuralHashes hashes first, so that others cannot see exactly which CSAM hashes they're testing against. Therefore they first need to decrypt their own database in order to get the corresponding hash value from which to derive the decryption key.
> Next, the client creates a cryptographic safety voucher that has the following properties: If the user image hash matches the entry in the known CSAM hash list, then the NeuralHash of the user image exactly transforms to the blinded hash if it went through the series of transformations done at database setup time. Based on this property, the server will be able to use the cryptographic header (derived from the NeuralHash) and using the server-side secret, can compute the derived encryption key and successfully decrypt the associated payload data.
Is that correct?
If so, then I agree that it is true that in the PSI system the server secret is completely necessary as part of the decryption process in order to decrypt the matching hash in the pointed-to location in the table. That being said, looking only at the information encrypted by the client, I don't think the server secret comes into play, right?
If I'm misunderstanding, and you're confident that an attacker would have to have the server secret to decrypt a photo (even if they already knew that photo's NeuralHash and were able to defeat the internal layer of encryption), then I definitely recommend posting a well-outlined answer to the Cryptography Stack Exchange, as that would be super helpful!
It’s not useful to talk about defeating the “internal” layer of encryption separately from the “outer” layer because vouchers should be stored as generated by clients at rest. The database leak should not include any unwrapped vouchers (that would be like using a password hash but storing plaintext passwords anyways).
But think of the children and security of the society. Couple that with constant monitoring of your car and you can be monitored anywhere
https://www.adfsolutions.com/news/what-is-csam
> However, the phrase “child pornography” is almost too sterile and generic to properly exemplify the horrors of what is being created. That is why many advocates, including the National Center for Missing and Exploited Children (NCMEC), believe this phrase to be outdated.
> NCMEC refers to these kinds of material as Child Sexual Abuse Material (CSAM), in order to “most accurately reflect what is depicted- the sexual abuse and exploitation of children”.
> As a result, many organizations and advocates now refer to this material by the new term rather than "child pornography" because it explicitly ties the material to the source of the problem; the abuse that is being perpetuated to create it. Furthermore, children are re-victimized every time a file is shared, sustaining the abuse in a continuous loop.
'Child pornography' is already too long to say repeatedly in a sentence so it is often shortened to "child porn." CSAM has far too many syllables so it too is shortened to two syllables in the form of the spoken acronym "CSAM" and so we're back to square one.
With that said this is a fascinating display someone pressing the reverse button on the euphemism treadmill. I don't think I've ever seen that before.
But children are incapable of being consenting performers. That's what separates abuse material from porn. So to make damn sure there isn't any overlap in the Venn diagram of media that depicts sexual acts, they'd rather not associate "child porn" with anything else that exists in the universe of "porn".
It's a completely separate category, not some "bad" end of a spectrum.
That's the reasoning I've heard before, anyway.
And I agree with you on the reverse treadmill thing. It's interesting. On a related tangent: I've always hated how journalists use the term "sexual assault" to refer to a wide range of offenses, from forcible rape to a passing grope. Although those are both bad things, it's clear that one is tremendously more harmful than the other. We should use language to clarify that.
I've wondered if there is an intent to distinguish between child pornography that teenagers are producing on their own to share with each other and the kind child pornography that pedophiles and child rapists create forcibly against the will of the children in the content.
Maybe that's the actual difference between CP and CSAM. Maybe both are a subset of CP.
I think saying “child porn” is clearer. That term is horrific as it is. If anyone sees that in a sentence they’ll be rightly revulsed and know what’s being talked about right away.
I had to have someone explain the CSAM acronym. How many people are going to skip over that because they assume its something benign and unrelated?
Using scarier words will get the public to trade liberty for security every time.
I've seen this argument many times, and I agree that initial act is horrendous of course, but I believe this is overstating the ongoing damage.
From what I recall of the episode, porn (rightly or wrongly) is seen as opt-in for the participants. But if children are involved it cannot be opt-in and is more appropriately described as rape or sexual abuse. Thus CSAM is the preferred term.
Should both parties be imprisoned and go on sex offenders registries for life?
CSAM provides a library with hashes that are compared against the photos on the phone.
The real problem comes when governments make laws that open the mechanism to different libraries of photos.
What happens when Apple, or the government, mandate an expansion of CSAM into detecting new material? Apple already has built a neural network to detect new explicit material...
Also, what happens when the USG mandates that Top Secret classified material must also be added to the database? Or when Russia mandates that homosexual pornography must be added to the database?
That doesn't appear to have happened in the 10+ years this kind of scanning has already been happening at various cloud providers - what specifically about Apple moving it down to the phone from iCloud Photos makes this more likely now?
For example, parents' photos of kids in the bath isn't CP. However _someone else_ having a _quantity_ of bath photos is CSAM, if they have no reasonable reason for possessing them.
We definitely should punish exploitation of children, including sexual, and we definitely should punish distributing images of this. But conflating exploiting, distributing, and viewing, and then putting a big taboo on this, is really not ideal. These things are different. Otherwise what we end up with is righteous frenzy when someone gets punished for sexting.
For example, the NCMEC database contains hashes for Nirvana's Nevermind cover. Completely innocent to possess within it's original and intended context.
I have not said whether I agree with this, because I do see problems when automating the process.
However, the precedent for it being used as a flag for law enforcement has already happened. Having a collection of similar imagery is considered CSAM - and that is likely correct. A collection is probably not innocent. But having one or two images may happen incidentally without your awareness of it.
As to who decides what constitutes significance? That is where you'll hit the most problems, and reasonable discussion of it will be quickly shut down with the same arguments used for automating a flagging system. The conversation requires nuance, but those currently calling for such systems aren't interested in a good faith discussion.
The process will never be fully automated, regardless of what they say. Cases will need to be reviewed. Things will need to be checked at some point.
They are trying to play the cog in the machine, that mechanically transmits information to law enforcement. But if we’ve learnt anything the last decade is that cogs are not impartial and can be very dangerous, if only because of the scale at which they operate. I can see several ways a user can face a kafkaesque uphill battle to prove their innocence. In several countries, just a child pornography case can be a social death sentence. And even a fraction of a percent of mistakes will mean millions of people might be dragged in this (let’s not kid ourselves: this is never going to stay in the US).
Personally I think (what Apple is doing) is misguided and ripe for abuse. I am very disappointed that they, of all companies, are pushing this nightmare.
> As to who decides what constitutes significance? That is where you'll hit the most problems, and reasonable discussion of it will be quickly shut down with the same arguments used for automating a flagging system. The conversation requires nuance, but those currently calling for such systems aren't interested in a good faith discussion.
Ultimately, a tech company has no business making this sort of decisions. This is something that needs to be sorted out by law.
Unfortunately, a nuanced discussion is very unlikely these days. Anyone looking not agressive enough will be pilloried.
Maybe they can also build an api so that governments can search easily for dissidents without the delays that the due process of law causes.
The best of both worlds: keep advertising their privacy chops to the masses, while also allowing any and every government agency a programmatic way to hash-verify the data passing through their systems in real-time.