There seems to be a vague idea floating around that this is built into the OS or the device just because the scanning happens on the device, but it’s not clear that’s the case. Apple doesn’t make the distinction between OS and app clear either.
Second, Apple almost assuredly will encrypt iCloud after this. So now we have the precedent of scanning encrypted messages. And that will then feed legislation that congress has been attempting to pass for years to kill any right to meaningful end to end encryption for messaging. https://blog.cryptographyengineering.com/2020/03/06/earn-it-...
This approach makes mass-sweeping of all server-side stored data harder to accomplish (whereas in, say, Google Photos, Google can break-glass server side to get into someone's private data, so they could hypothetically do a mass-scan if the government demanded it).
First line of their privacy policy is: “Apple is committed to your privacy.”
This is what I meant by mixing up (and blurring the lines between) app-level and OS-level capabilities. It might not actually be mixed up technically, but it seems to be the user perception.
The only thing stopping your phone from keylogging your password to a server in the NSA somewhere if it recognizes a specific trigger pattern is Apple's willingness and ability to resist pressure from the US, etc.
Think of what would happen if you tried to make your average Silicon Valley dev team design, implement, and test a surveillance system they didn't want to build and that was immoral. They'd resit in an infinite number of ways that would delay the project virtually for ever. Short of summary executions, I bet you could not get a nice, efficient, effective system.
On the other hand, once the dev team has enthusiastically built the system that scans for any image, it's entirely easy to say "Now, make it look for these images." They have no avenue for resistance other and a up front no. And a government that wants to do totalitarian things knows many ways to force a yes.
In general, a company at that size would approach this problem by figuring out who in the company is willing to take on an unsavory challenge like this and then forming a skunkworks out of them, slightly sequestered from the rest of the company.
I'm not saying Apple has done it, or that they're incentivized to. But it's trust-turtles all the way down. Either we trust them to say "No, you can't use our tech to harm our users," or we don't.
It wouldn't be that. It would be defense contractors sitting at Lockheed or a few blocks from DARPA whose daily bread is making a Tech Sandwich whenever the Broad Agency Announcement for one shows up on sam.gov, or on the DARPA page, or the variety of procurement sites that the government doesn't expose to the internet. If they want it, they can get it -- no persuasion of liberal tech-bros needed.
Updated original comment.
Someone was worried about how they handle the keys. They have solution for that already: https://blog.cryptographyengineering.com/2016/08/13/is-apple...
This is false. They present a web interface showing the photos. The UI isn’t locally generated entirely using JavaScript to decrypt the data. They only way this can happen is if Apple has the decryption keys.
iCloud Photo Library has never been private. Apple has always been able to view your photos.
https://9to5mac.com/2021/08/05/report-apple-photos-casm-cont...
Apple has the keys; the data is encrypted at rest and in transit, but they can be compelled to use them.
(Caveat that if you have iCloud backup enabled - which it is by default, the backups aren't end-to-end encrypted. This feature is basically on the convenience side of convenience vs privacy / security - too many consumers would irretrievably lose their data if iCloud backup weren't enabled by default)