Gmail will call the cops on you based on the content of your emails
pluralistic.net
pluralistic.net
The title is incorrect (as the article itself notes) as neither Gmail nor email contents were the trigger; rather, the situation unfolded as a result of Google Photos automatically uploading a pediatric medical photo, flagging it as CSAM, and setting in motion a law-enforcement reporting process.
I think this represents two issues. One: AI is not AI but a litany of conditionals that poorly reflects what we as humans can determine fairly quickly. Unless the process is unquestionably objective and possibly quantitative in its outcome, I just don't feel full automation will be without these events.
Google as a business automates everything. They are over their ski-tips in the amount of "getting it perfect" that is possible and are completely accepting of "close enough". I don't think this thought process is unique in the corporate world and unless revenue is impacted heavily, they will not be incentivized to incur the heavy cost of making the end-user whole. The majority of revenue is ad based meaning B2B. Users are a bucket of data, not the person they are selling to.
It's like complaining that a cattle farmer doesn't treat his cows kind enough...the farmer would think you're nuts despite the masses possibly agreeing.
As it relates to this specific type of content, I'd rather see something bad happen to good people than something good happen to bad. Maybe it's unpopular, and I'd hate to be the person on the receiving end, but unless we can have truly objective AI, I don't think we will be absent of these types of problems. In the meantime there has to be someone in the background fixing these cases and Google has shown clearly with YouTube...unless you are making enough noise they aren't listening.
You don't need "objective AI." In this case, you just need a process that goes to a human when it is contested. Sure, make the user pay a certain amount of money to get an investigation by a third party or to have all the information provided to a court. But if Google made a mistake, and that mistake caused tort, Google is going to have to pay out.
The appropriate thing to happen here is something along these lines:
Google's AI tells the person their account is cancelled, but they can contest it for $100. The person contests it, and a human is put on the case and investigates. That person determines that Google made a mistake, and Google will restore the account quickly, and pay the person $200 for the hassle.
And by the way, while I'm sure Google should have to restore the account in this case, I'm not so sure Google should have to compensate even for the investigation fee. Because honestly it is a tad clueless to take photos of a kid's privates and allow them in the cloud. He didn't do anything truly "wrong" (in the molestation/ child porn sense), but still should have known it wasn't a good idea, so paying $100 for his mistake and being without his account for a week sounds about the right "punishment."
It is by definition distribution of this material and to have software edge-case exceptions allowing certain situations through is something I can't imagine anyone willing to sign their name to endorse.
It seems from a heartless management perspective that the simplest decision is to walk away from the whole situation, wash their hands of it, and accept this as collateral damage.
I meant to say specifically that until we can define things from an objective and not subjective stance, AI will not fare better.
That's an interesting proposition on the solution. I feel I see some potentially undesirable effects similar to the legal system but perhaps better than what exists currently.
Whoever is responsible for this decision and the resulting communication, I have nothing constructive to say. Sincerely, fuck you.
I can't believe they went through this person's account to find justification for their first mistake. This is intolerable.
> Google says they won't give Mark his account back because they found another "problematic" image in his files: "a young child lying in bed with an unclothed woman." Mark doesn't know which picture they mean (he no longer has access to any of his photos), but he thinks it was probably an intimate photo he captured of his son and wife together in bed one morning ("If only we slept with pajamas on, this all could have been avoided.").
A bunch of people need to be fired at this point in the story.
The world is very complicated and our ML models aren't great at dealing with OOD or infrequent samples. Mistakes are going to happen, and this is okay, but we need to ensure that we have mechanisms that can quickly remedy these mistakes. There are countless people who have had far less recourse because their stories did not make international headlines. Your ability to go viral shouldn't highly correlate with your ability to solve mistakes.
No email -> no banking -> no rent -> no house -> ??
Gmail won't chase you and throw you in a cell if you run from them. The police will.
Gmail being one of the worst. I've been using my own domain + fastmail for years and from time to time I still find stuff tied to my old Gmail account. Right now I have forwarding set, but if it got disabled overnight I would still run into issues.
All the rest I can survive without, but that took some doing. And I started years ago.
Google photos is easily the best product I cannot use - maybe TikTok is a close second. But sometimes it’s worth being that stubborn nerd and fighting for an alternative - even if the alternative is a distant second.
It's a Raspberry Pi that is ready to go 'out of the box' so you can install software immediately rather than fiddling with it.
I believe the PiBox OP linked is intended to be a PiBox that comes pre-installed with their software, Photostructure, so you can run your own instance easily without fiddling with a Raspberry Pi.
...and yet the page for that pibox product makes no mention of photostructure at all, let alone saying it's pre-installed. I'm just trying to point this out. The linked page would appear to be a complete non sequitur if not for the fact that I can see '/photostructure' in the URL.
PhotoStructure installs via a couple clicks in their app store, though--it's super easy. Barely an inconvenience.
(FWIW I'm the author of PhotoStructure)
Yet. They will be forced to implement it.
Your photos and videos never touch our servers, _by design_.
I've worked for several large tech companies that hosted personal content, and seen how _utterly miserable_ it is to try to comply with the morass of local customs and laws.
I have no willingness or desire (or sufficient revenue) to spin up a safety and security team: PhotoStructure will always be entirely self-hosted.
Because PhotoStructure is always self-hosted, the onus falls on the user to comply with any local laws, just like any other software tool.
That said, my security.txt has a canary.
Some busy body at google acting as morality police over what they see as "problematic" behavior.
Time to backup my Google content and get a dumb camera.
The police said that Google "contacted" them, which could mean an Email, and online form submission, a paper letter, etc., not necessarily a phone call.
To “call on” someone, maybe. But this says “call the cops”.
Maybe I’m just getting old, but anytime I’ve heard anyone say they “called the cops”, they dialed them via phone.
With that in mind you can decide if the convenience is worth the risk of losing it all. For some things it will be but not for others. You should ensure that everything you cannot afford to lose has at least one backup that you fully control.
Well there is something you can do about it, which is involve courts and/or legislation.
I mean your advice may be good for individuals who haven't (yet) run into this, but I think the discussion should be more "why is this acceptable and what can we do about it?"
If an account is terminated, it is done in a way it can be brought back if the user thinks it was unwarranted.
The user should be told that if they want to contest it, they need to pay a certain amount of money to a third party, to pay for the time of the people who do investigate. ($100?)
Those investigating it are not owned by or controlled in any way by Google. Those investigating it are allowed to rule not only that the account be restored, but that the user is compensated (maybe up to $5000).
If the investigation doesn't go the way the user hoped, they should be able to use the courts and Google should be ready to make all available data available.
On the bright side, if you own it, at least an AI can't shut you down and ruin your day.
Even a barebones email host is out of reach of most people.
The question in my mind is as to what the limits of this snooping is? Discussing out of state abortions in a location where it's outlawed? Conducting a homosexual relationship in a repressive state?
I get that we have our ideals here, but our legal system does not necessarily lead to justice - especially when asymmetric imbalances are in play.
IANAL but I would not lawyer up against a huge corporation like this and expect to keep my money.
> Correction: An earlier draft of this story misstated a technical detail; Mark didn’t email his photo to his doctor; rather, he took the photo with his phone and the image was automatically synched to his Google Photos account, triggering a scan.
A more accurate title is: Google will contact the police based on their analysis of your Google Photos.
If your photos are being scanned the system is compromised. Perhaps the author intended to produce a chilling effect necessary for people to understand this. There is no reason to believe your entire life on the Google platform is compromised and available (freely!) to any government agency that wants it.
Unfortunately, people won't realize how dangerous this is until a skin tone photo of something entirely benign gets marked as CSAM by a robot, and you wont know until the police are letting you know they want you in for questioning. What a dystopian world we live in.
Mixing up text and images isn't really pedantic. One is reading my thoughts, one is monitoring my captured moments. They're both bad, but very different. Even the author issued a correction at the top of the article, but should have updated the title to reflect the correction.
> However, it's not a far stretch (nor hard to imagine) this same technology is running on every Google-enabled service.
OK, but then write those articles, with proof. Until then, just keep journalism as accurate as possible. When we lower the bar so much that we're writing about things that we can imagine being true, it's just fantasy land.
> If your photos are being scanned the system is compromised.
I can have a Gmail account without having a Google Photos account, so no not everyone who sees the "Gmail will call the copy on you" headline has a reason to panic.
> Perhaps the author intended to produce a chilling effect necessary for people to understand this.
That's really irresponsible fear-mongering and I doubt was done by this author since they were responsible enough to post a correction at the top of the article. The truth is the scariest thing because it's real. Fantasy isn't scary because it's fiction.
He also said that even if one does that, it's likely that all of the equipment that was ever connected to the device will be confiscated along with it, and might not be returned for a very long time. I.e. if you run a small forensics shop, have at least two of each piece of hardware in different locations so the confiscation doesn't put you out of business.
So while I feel like Google's reaction was extreme, and they should have unlocked his account after being provided with confirmation by the doctor, I suspect they were just doing their best to comply with US law.