Apple clearly documented that this was only for iCloud uploaded photos, and indeed the technical description makes clear that this is only designed to work with uploaded photos: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni....
Not sure why they would need to do that. It does open up your phone for scanning and uses beyond what they initially layout.
The whole point of the protocol (as described in the Apple whitepaper) is to allow clients to attest to perceptual hash matches without the server having access to the plaintext.
So, the irony of all of this is that the Apple design is effectively more private than the status quo, but everyone freaks out about it.