This is wrong - the iCloud check is against known CSAM hashes, the false positive rate is essentially zero.
The idea is to take an image and have all of its possible derivatives create the same hash.
For example, if a hash was made of the Mona Lisa, any copy no matter how large, small, black and white, would have the same hash.
Think of all the ways the Mona Lisa could be transformed and still be the Mona Lisa.
The combinations of ones and zeros would be in the billions. If not more.
And all those possible combinations of ones and zeros go back to the same “hash.”
That’s extremely resourceful intensive.
My guess is that they are going to transform the images into a very low resolution, black and white thumbnail. Then compare it against known abuse images that have been similarly transformed.
Or they’re using AI. They might be using AI.
Either way, it’s guesswork. How many images might be transformable to the same black and white thumbnail. I don’t know.
If the system was matching against known cryptographic hashes the collision / false positive rate would be small, but the fuzzy matching involved with perceptual hashing necessarily has a greater false positive rate.
And that doesn’t even begin to address the detection of sent and received “explicit images” which are detected on device and don’t have a set of known hashes.
The iMessage bit is different - it's only on device, only on child accounts, and only alerts parents. It's more akin to a parental control feature than anything else.
They have nothing to do with each other, and I've seen a dozen people on HN confuse them. If the message is getting muddled here, it will be hopelessly conflated in less technical circles.
I'm concerned and upset about the CSAM filter for all the reasons that keep hitting the front page, but don't care about the opt-in parental controls at all, and if I had kids, I might want them.
But if I thought the CSAM filter worked like the nudie-detector filter, I'd be wigging out.
Bugs in the application of the code, combined with human complacency and mistakes can certainly lead to errors, even if the cryptographic algorithm itself was perfect.
We really need to bring back comp.risks
So now instead of sending just one nice innocent very high resolution images of "Tokyo City" or something with something horrific hidden somewhere you have to send a few such images.
That is reassuring. I can never believe anyone except me will think about that.
(If the system is too dumb to detect this it is worthless, and if it is smart enough this opens the floodgates for anyone wanting to make trouble for just about anyone.)
suppose you have a partner who is a 'petite' woman of 34. She enjoys posting nudies on a website, but without her face in that picture. Someone who collects child porn downloads it, because he enjoys that picture. A year later he gets caught by the police and all his pictures get marked as 'verified child porn'. Suddenly you get marked as owning child porn.
https://www.nytimes.com/interactive/2019/09/28/us/child-sex-...
Apple’s thing has some sort of threshold anyway so one image would not trigger it. I don’t buy your example - the CSAM images are not what you’re describing.
They are instead fuzzy classifiers, and thus have non-zero error rates.
How can you do that, considering md5 can have collisions?
For a hash (whether cryptographic or perceptual), there is a chance of random collisions and also a difficulty factor for adversarially-created intentional collisions. The random collision probability has to be estimated based on some model of the input and output space (with cryptographic hash functions, you would usually model them as pseudorandom functions and assume that the collision probability is the same one created by the birthday paradox calculation).
Intentional collisions depend on insight about the structure of the hash function, and there are also different kinds of difficulty levels depending on the nature of the attack (preimage resistance, second-preimage resistance, and collision resistance). Gaining more insight about the structure of the hash function can act to reduce the work factor required for mounting these attacks. That should be true for perceptual hashes just as much as cryptographic hashes, but presumably all of the intentional attacks should start off easier because the perceptual hashes' threat models are weaker and there's much less mathematical research on how to achieve them.
And in AI systems involving classifiers, it was generally easy for people to create adversarial examples given access to the model. Perceptual hashes for estimating similarity to specific known images aren't the exact same thing because it's less like "how much like a cat is this image?" and more like "how much like NCMEC corpus image 77 is this image?", but maybe some of the same techniques would still work. In the cryptographic hash analogy, I guess that would be like trying to break preimage resistance.
To mitigate adversarial false positives one idea is to use the combination of a cryptographically strong hash along with a randomly selected perturbation of the file. Prior to hashing, perturb the file and submit both the hash and the selected perturbation to apple. Apple selects the DB based on the perturbation and proceeds with matching and thresholding.
If the attacker does not know how the image will be perturbed prior to hashing then he cannot generate an image which matches with known CSAM.
I think the rest of us have been discussing how this can and will be abused, by definition by adversaries.
Many of us have also observed for years how systems are abused so we sadly have a gut feeling for this.
So when we have a billion iPhones in the wild taking 10 images a day...1 in a trillion chances happen every few months. Now, if that triggers some further review, maybe that's an acceptable false positive. If it triggers a SWAT team, I don't think it is.
I highly doubt that
Based on the fact that ultimately we can't check the system. And based on the fact that at some point in the chain humans are involved. [1]
What we are left with is "trust in Apple" not "trust in math".