This site demonstrates no danger at all. Hash collisions like this are expected.
Anyone who claims this is evidence of danger either doesn’t understand how the system works, or is deliberately misleading people.
Anyone who claims this is evidence of danger either doesn’t understand how the system works, or is deliberately misleading people.
Here is the relevant paragraph from Apple’s documentation:
“as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.”
https://www.apple.com/child-safety/pdf/Security_Threat_Model...
Does this mean, that they have another perceptual hashing algorithm, and this is happening purely on the server side?
Because that could be quite genius. It would be really hard to adversarially fool two different algorithms.
The arguments have been stated before, the principal one being that scanning the users device goes against the wish of users and what Apple until now has stood for.
I think Apple fails to see the seriousness of this.
But i don't consider it E2EE if you are scanning content on the device first.
Apple has a billboard right outside my office. Up until last week it said "privacy" in big letters. Now there's no mention of privacy at all.
The comment chain I was responding to is about the claim that neuralhash is broken so the system is dangerous.
This claim has been repeated multiple times in this forum. Indeed it is one of the loudest two arguments against the system and is simply false.
The sum of elements which are false is still false now matter how many you have.
It’s certainly not universally true that scanning the device goes against the wish of the users. There are many users who are quite happy to have their devices scanned if it is part of making life harder for child abusers. I have spoken to such people.
Therein lies the problem. This system really doesn’t do anything harmful, and it really does just make the tool less useful for collecting child pornography.
There is no simple argument against it, hence all of the misrepresentations about its purported technical problems.
I strongly dislike the feature, and would rather they don’t do it.
My arguments are that I don’t want to be, even in principle, suspected of something I haven’t done. And that trust should be mutual. If Apple wants me to trust them, I want them to trust me.
I don’t see how we help make the world better by using false narratives about technical issues to get what we want.
1. There is a possibility, however tiny, that an innocent image will still match both hashes. The probability is not zero. It can't be. They're hashes.
2. By telling people that there is no chance of an innocent image matching both hashes you are forcing the burden of proof on to the victim. If someone is unfortunate enough to have an image that matches both hashes they will be dragged into a law enforcement office and told to explain "why the foolproof, perfect, impossible-to-cheat system said there was child porn on their phone". It will be on them to explain why the system isn't correct. The presumption of innocence is lost when too much faith is placed on technology.
That is why this is dangerous. Arguably it's a well-designed system that safeguards children and catches despicable criminals, but unless people understand it isn't infallible, and stop arguing that it is, then it could cost an innocent person their freedom. That's a high price to pay.
"Unlikely possibility" shows either blatant facetiousness or blatant ignorance. Who couldn't predict the explosion of collision-generating projects?
Yeah it's not like there are a lot of people on the darknet that do illegal things all the time. /s
You only need one of them to compute the neural hashes of the images and then upload them. For others, it is not illegal to have or distribute the neural hashes.
How many of them are developers with enough knowledge compared to all developers in world with enough knowledge? And willing to touch CSAM material?
I think there is a quite difference. And they have better things to do than pranking some people. Because you are not getting jailed from hash collisions in the country with working judicial system. You have bigger problems if judicial system is not working.
As other comments note, that's not a huge increase in difficulty for an adversary willing to deal in actual CSAM.
Isn’t this exactly like that? Or am I missunderstanding something.
https://twitter.com/ghidraninja/status/1428269674912002048?s...
Quoting the tweet for those who dislike twitter:
> The procedure for this was pretty simple:
> - Started with a placeholder image
> - Got the hash of it
> - Changed text of the image to match the hash
> - Then used
> @anishathalye great neural-hash-collider which took only 1-2 iterations
N.B. it starts with 'started with an image, and the hash of it'
[0]: https://twitter.com/ghidraninja/status/1428270675010199554
Anyone who claims this is not evidence of danger either doesn’t understand how the system works, or is deliberately misleading people.
Fixed it for you :)
On device scanning did not exist before apple. Apple is doing it. Now devices don't have your best interest in mind. Bad precedent, easy to be abused in future by others with even worse implementations or motives. Evidence of danger.
Their hashing algorithm by nature of being a hashing algorithm will have collisions. That is unavoidable. Unavoidable false positives are not great in the context of your device ratting you out to the authorities.Evidence of danger.
QED.
Virus scanners scan on devices. Practically all big brand commercial scanners nowadays scan and check findings on-the-fly online.
That said, Microsoft ships Defender since XP or so. Apple has some basic malware scanning since Catalina where it compares hashes of executables.
Facebook checks hashes of pictures sent in messages against hashes of abuse picture since years.
> Their hashing algorithm by nature of being a hashing algorithm will have collisions.
Yes and no. You'll probably not find any collision in a SHA-3 hash. Quite likely you won't even encounter a SHA-1 collision. But this Neural Hash? It's been published since this short amount of time and already such a collision is being presented.
As a user I probably don't want Viruses, Ransomware or anything with comparable destructiveness on my computer. But at the same time the implementation should be rock-solid and user-controllable. That's a no-brainer for anything sha-based but for Neural Hash it isn't.
If these scans warned you that the content was questionable and quarantined it with an option for you to release it on your own review, it would be truly analogous to antivirus software. However, that's not what's happening. I think a lot of the arguments against are overstating certain elements and often ignoring others (like that there are two manual reviews before anything goes to law enforcement), but we can't go in the other direction to defend the system either. It is not like existing on-device scanning solutions that are meant to protect (by some definition) the user or the device.
> Unavoidable false positives are not great in the context of your device ratting you out to the authorities. [emphasis added]
If the detection were immediately sent to authorities, this would be an issue. And it's possible in the future (or in practice) that the manual review could be removed or will be a rubber stamp. However, they do have a manual review process slated for this. The reason I promote caution with regard to this element of your argument is that if you have an easily rebutted statement, people will disregard the rest. It's why hyperbole in most political arguments should be strongly avoided, it makes it easy for people to dismiss the rest of the statement(s) no matter how valid.
Yes, exactly. They themselves state this, and have a mechanism for preventing collisions from turning into false positive matches.
I have to assume you simply didn’t know that.
Apple have said that if you have x amount of hash matches/ collisions to a set of images you'll be flagged to the authorities?
This site has proven now that it's easy to create pictures whose hash matches (colides!) thus potentailly flagging lots of poeple
0: https://pocketnow.com/neuralhash-code-found-in-ios-14-3-appl...
1: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
But no-one, yet, I think, has cracked open the blob containing the neuralhashes of the NCMEC CSAM corpus which are the actual hashes you need to create pictures to collide with to flag people under Apple's new system, right?
(This may be because the blob isn't yet on the phones; I don't know, I just haven't seen anyone mention it wrt creating hashes.)
Risky. But ok, you still need to get hold of 30 images[1] that are in the NCMEC corpus before it'll even get flagged up to a human reviewer.
[1] Do we know if it's 30 distinct images or will 30 copies of the same one trigger things? The latter is far easier than the former for the gathering step here, of course.
Collisions are expected. Being able to create a preimage on demand is not.
The entire scheme risks running afoul of Goodhart's Law (https://en.wikipedia.org/wiki/Goodhart%27s_law). By calculating the neuralhash of an image in a publicly-known way (so the calculation can run on the user's device), someone distributing illicit material can modify said material to deliberately collide with a known-good image.
This provides a reasonable guarantee that the illicit material is not in fact caught by the on-device scanning, and if the illicit material is later added to the "known-bad" database the collision with the known-good image will decrease the signal/noise ratio of the whole monitoring scheme.
It's photodna, people could make images that match both.
You really can't say anything about "unlikely" when the other algorithm is a secret. Perhaps it's just "return true". Apple failed to disclose[*] that neuralhash could be adversarially matched until the public exploited it, what else are they failing to disclose?
[*] I say fail to disclose instead of didn't even know because I believe there is just no way they didn't know this was possible. I generated my first match with almost no experience with the machine learning tools in about an hour, most of which was spent trying to figure out how to use python neural network glue stuff. It should have been expected by design, and other that getting lucky with some parameters basically the dumbest thing possible works (not for every image pair, but enough).
That seems like a statement that needs to be justified. I suppose one could apply a 50% dropout, construct adversarial pairs on that network, and then run them though the the other half-network to get an idea of the frequency of match on both.
My intuition is that the probability of match of an adversarial pair on a second, independent network should be close to the probability of a non-adversarial collision. Robust adversarial perturbations are hard enough when only considering class labels, nevermind hidden state
You’re obviously not being honest. You don’t believe it could actually be just ‘return true’.
> [*] I say fail to disclose instead of didn't even know because I believe there is just no way they didn't know this was possible.
Of course they knew it was possible. That’s why the system doesn’t rely on it not being possible.
If you're going to question my honesty, can I ask what financial interest you have in apple or this abuse system, if any?
> Of course they knew it was possible.
I'm glad we agree. But the fact that their hash is subject to adversarial images, causing totally innocent images to be decryptable by apple, is disclosed nowhere in their documentation or security analysis.
I think that is extremely concerning, particularly because it undermines a central plank of their claimed protections: They claim they will protect against state actors including non-child-porn material in their databases by intersecting multiple databases from multiple states. This claimed "protection" is completely obliterated by state actors being able to modify child porn to match the hashes of other images and then submitting this material to each other.
The questioning is a response to your comment.
Do you seriously think that Apple’s second hash function could be ‘return true’?
Firstly it’s a dishonest response since we were talking about how the code worked. But let’s put that aside and consider it anyway.
If the code literally returned true, the second hash would have no effect at preventing collisions for any user at any time. As you say, for this to be the case, Apple would need to be lying.
A subpoena wouldn’t be issued without cause, would only impact specific individuals, and would have no impact on the general protections provided by the secondary hash at preventing the adversarial attacks proposed.
Nothing about this is even close to functionally equivalent to ‘return true’.
But the website correctly points out a possibility of "[m]isuse by authoritarian governments" and it is what we should be definitely concerned of. It is not a CSAM detection algorithm. It is a generic image detection algorithm.
[1] My own analysis: https://news.ycombinator.com/item?id=28223141
I’m actually tempted to delete my account and stop participating, but I don’t know where else to go to instead. I liked old HN.. anything like it now? Suggestions?
There was a commenter here a little while ago making wild claims that he was being monitored because he used a ProtonMail account to sign up for a government service. He was being taken seriously by a large number of replies.
I'd love to see/find an alternative as well, but I think part of the issue is just volume. You'd need a platform that's restrictive by design to keep it from getting over populated and landing on the wrong side of the quality-quantity curve. I'm not sure that can exist in a free (as in beer) platform.
Part of me is worried that there aren’t such places, in part because culture wars have made the online world so toxic.
However I agree that this time it seems different. I don’t know if it is because this topic is genuinely more complex than than most. I suspect that is part of it.
But please suppose most of the readers here do understand how the system works, and most of them don't approve how it works.
- There are two main components here: automated flagging and supervised, human confirmation
- It has been proven that the automated flagging system can be fooled
- The last line of defense is now a human: they have to look at an obfuscated version of your colliding image to determine if it's objectionable
The current discussion is mostly around the second portion, moreover what kind of social-engineering attacks would be most effective to overcome a relatively flimsy single-human point of failure.
This statement is absolutely false. No such proof has been presented anywhere, including in the linked article.
Adversarial Neuralhash collisions are expected and the system is designed avoid false positives if they are encountered.
Here is the relevant paragraph from Apple’s documentation:
“as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.”
https://www.apple.com/child-safety/pdf/Security_Threat_Model...
This is incorrect. You need the actual NeuralHash for the collision attack to begin with, but the device only has blinded hashes so you have no hash to begin with.
To be fair, though, those two steps have failed -in sequence- enough times for it to be statistically significant on the "OH WTF" scale.
You are also leaving out the step where they run the scaled down image against another separate app that isn’t susceptible to this hack since the output isn’t known.