Comment by buildbuildbuild | Hacker News Reader

Comment by buildbuildbuild | Hacker News Reader

Parent Full thread

buildbuildbuild

I thought about this, but you're still stuck trusting the implementation unless you test with actual illegal data, which is often criminal and immoral to obtain.

Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction)

Testing by using a test phrase, like in a spam context, is inadequate here as a scanning system can trivially be architected to pass those publicly-known tests, while still overreaching into people's personal files and likely miscategorizing content and intent.

If a user connects via a VPN to Russia for whatever reasons, does their personal content start getting reported to that country's law enforcement by their notion of what is illegal?

Parents often have all sorts of sensitive photos around which are not held with exploitative intent. "Computer says arrest."

I always imagine aliens hearing about something like this and being stunned.

"How can data be illegal?"

"There are bad things, but how can you decide what is bad and show how it's bad without examining and discussing it?"

You can only go so far merely alluding to things. Somewhere the rubber has to meet the road and you have to have concrete data and examples of anything you need to study or make any sort of tools or policy about.

It's like parents not talking to kids about sex. You can avoid it most of the time because decorum, but if you take that to it's extreme you have just made your child both helpless and dangerous through ignorance.

Somewhere along the way, you have to explicitly wallow directly in the mess of stuff you seek to avoid most of the time. That "seek to avoid" can only ever be "most of the time". It's insane and counter-productive to try to see that "most of the time" as an incomplete job and improve that to 100%.

I guess in this case there will eventually be some sort of approved certified group. A child porn researcher or investigator license. Cool. Cops with special powers never abuse them, and inhibiting study to a select few has always yielded the best results for any subject, and a dozen approved good guys can easily stay ahead of the world of bad guys.

> Example: How does a researcher test whether algorithmically-classified illegal imagery stored on user devices is being scanned and reported home to Apple's servers, and what those bounds of AI-classified criminality are? (presumably with respect to what is illegal in the user's jurisdiction)

I'm not an expert in AI so this might be totally off base but I feel like you would be able to use an "intersection" of sorts for this type of detection. You detect children and pornography, the children portion trains it for age recognition and the porn portion trains it to see sexual acts. Slap those two together and you've got CSAM detection.

Possessing CSAM if it’s for abuse-prevention is not immoral, regardless of what the law says. Saying otherwise is a slippery slope to saying judges, jurors and evidence custodians are also immoral. In fact if possession is so immoral, CSAM trials shouldn’t even have visual evidence. We should just trust the prosecutor.

while i vaguely agree with the idea behind this comment, this explanation is particularly poor. by that logic, it should be allowed to kill people to prevent murder. it is, in fact, allowed to kill people to prevent murder, but only in specific legally-prescribed circumstances. typically, only specific people are allowed to kill, and only to prevent an immediate murder. the same applies for child porn: it is allowed to have child porn to prevent child porn, but only under certain legally-prescribed circumstances.

Reply on news.ycombinator.com