I’m much more worried about adversarial attacks rather than accidental false positives.
For example, a year ago I was trying to track down how some unscrupulous photos ended up in my photo library. I finally realized that Telegram had, without my knowing permission, been adding all photos I had received in a crypto discussion group to my Apple photo library.
There seem many ripe opportunities for targeting.