Hacker News Reader

Comment by Latty | Hacker News Reader

When I worked on analysis software for police & intelligence agencies, it quickly became clear that no matter how much you present something with clear "this is a heuristic, it can give you leads", a scary number of people will take it as "this will tell you whodunit" and just assume it's infallible.

If people are going to treat these things like they are forensic evidence and not lead generation, then they shouldn't be legal. Some precedent that people need to check these things before acting on them would be very good.

View on HN

From The flaws of policies requiring human oversight of government algorithms https://www.sciencedirect.com/science/article/pii/S026736492... :

    human oversight policies provide a false sense of security in
    adopting algorithms and enable vendors and agencies to shirk
    accountability for algorithmic harms.

And the better The Algorithm is, the less people check, and the more of its faults pass through.

Having worked on systems that required human oversight, I've seen first-hand, and analyzed data on, how people click the "LGTM" button without reflecting. We even noticed how when we improved our predictor with expert input, so that it performed better according to experts who worked with it, it would perform worse when we benchmarked on the previously "manually checked" data set. Analysis showed that the manual checks had let through so many of the algorithm's faults, that when the algorithm improved, it got a worse score.

    We present seven flaws of predictive optimization. Our aim is to outline a set of
    objections inherent to predictive optimization that cannot be easily fixed using a
    design or engineering change. Taken together, these critical flaws undermine the
    legitimacy of applications of predictive optimization.

    1. Good predictions may not lead to good decisions
    2. It's hard to measure what we truly care about
    3. The training data rarely matches the deployment setting
    4. Social outcomes aren’t accurately predictable, with or without machine learning
    5. Disparate performance between groups can’t be fixed by algorithmic interventions
    6. Providing adequate contestability undercuts putative efficiency benefits
    7. Predictive optimization doesn't account for strategic behavior

True, "check" was poor phrasing, because these things don't produce evidence, they produce alerts to guide investigation.

It should be a given that these systems don't produce anything actionable, they are a guide for a user to look more closely, of course, that requires someone in the chain stops and says "this isn't usable evidence".

Very interesting - thanks for giving us the benefit of your experience.

In the case of store workers, it's not obvious what other checking they could do. The only reasonable use I can see is for them to keep an eye on the person that has been flagged. But I expect that the entire purpose of this tech is to reduce the number of store guards they need to pay, so they are likely not to have the manpower. If the policy is 'don't let them in the store', I don't see how a store worker can avoid making false accusations.

There's something fundamentally wrong with these probabilistic analysis tools being sold currently as "artificial intelligence". The nuance in "probabilistic" is quickly lost and the output is taken as deterministic.

Any piece of software that puts out data that is acted upon should have clear disclaimers attached. Calling it "intelligent" is irresponsible.

"Calling it "intelligent" is irresponsible." How so ?

If a human of animal behaving the same were to be describe as intelligent, than so are those systems.

How so? People take it at face value and harm other people based on its outputs, as per the article linked.

> If a human of animal behaving the same were to be describe as intelligent, than so are those systems.

I cannot parse this sentence. Behaving the same as what? What bearing do claims of human or animal intelligence have on me stating that generating liability by trusting probabilistic analysis, and actively harming people, by calling software "intelligent", is not responsible behavior?

we can call it frediness, but the problem isn't what it's called, the problem is that the profit motive only works when there's competition. if there were several shops nearby and one of them started rejecting 80% of customers based on frediness, they'd go out of business and the others who prefer having a human guard would prosper. if it's the only shop within a 10 mile radius, and it only flags out 1% of innocent people as shoplifters, then the profit motive doesn't provide the balancing force. that's what regulation is for.

The bearing is on the claim that calling software "intelligent" have anything to do with trusting or liability. Dumb or not, no one is supposed to blindly trust it.

I can think of one possible regulation that I think would produce the proper behavior incentives.

Require that the system intentionally produce false positives (selected at random) at approximately the same rate as the true positives. This means that the users of the system will be aware that any indication has only a 50% chance of being "real", and therefore can be used as an indicator to increase caution but NOT on it's own as a signal of guilt.

Why not just flip a coin then? It would be much cheaper?

I imagine the solution space is much bigger.

I think it could be an actually pretty good solution for this. A bit like Chaos Monkey but for human systems. Introducing error often enough that operators don't assume it never happens.

If the expected rate of known shoplifters is 1 in 10,000 then the human security team can keep a close eye on 10,000 people (one of whom is probably a shoplifter) or on just 2 people (one of whom is probably a shoplifter). The latter is much more efficient.

Without the intentional false positives they could simply throw out 1 individual with no need to keep an eye on anyone at all. That's enen MORE efficient, but it utterly ruins the lives of anyone falsely identified.

they'll just throw both out.

Similar to story points and task estimates suddenly becoming deadlines

>If people are going to treat these things like they are forensic evidence

Are most kinds of forensic evidence actually any better?

Well, plenty of bunk stuff (bite mark analysis, hair analysis, etc...) has been used in the past, but in theory anything used today should be backed up by significant evidence and rigorous methodology, and obviously it'll vary based on jurisdiction, but in theory courts should require that background.

Certainly, more transparency than "we bought some facial recognition software, no, you can't see the code, that's a trade secret".

> no matter how much you present something with clear "this is a heuristic, it can give you leads", a scary number of people will take it as "this will tell you whodunit" and just assume it's infallible.

That's precisely my anecdotal experience.

Subtlety and nuance are easily ignored where their absence appears to provide easy answers and the avoidance of responsibility for their outcomes.

Similarly to "It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It".

I don't trust a sales person or marketer to not hype such a system as being flawless whodonit machine. I certainly don't trust that the people purchasing these machines won't communicate to their staff that it's a flawless whodonit machine.

That's because the step from "potential leads" to "flawless machine" is a relatively small leap easily hidden by throwing out bad stats and case studies about people caught red handed. Even if you say the machine is 80% accurate, a lot of people will happily accept harming the 20% on the chance they might be in the 80%. (Though, I'm willing to bet the false identification percentages are both not public and not kept up to date, if they are even known).

Reply on news.ycombinator.com