Hacker News Reader

Comment by handmodel | Hacker News Reader

>identified the individual as Ousmane Bah based on the examination of the temporary learner's permit he is said to have had on him

So essentially a kid stole another kids ID. Because the (thieving) kid had gotten in trouble with the same ID before it was also a match in the database.

It does seem like a true failure but the headline makes it sound like this was an AI falsely accusing someone based on pure randomness. Even without the tech, the police would have had the name of the wrongly accused person because the security guard only had the fake/wrong ID and the person's ability to match faces was not great.

View on HN

> the headline makes it sound like this was an AI falsely accusing someone based on pure randomness.

no, it doesn't. the headline makes it sound like facial recognition is unreliable for some reason.

the reason, in this case, is that the Apple store and/or SIS personnel gave the recognition system Ousmane's name when they weren't able to prove it was him, and the document provided for identification was not actual proof of identification.

Further facial recognition matches assigned further blame to the misidentified person, increasing the evidence against him.

Then, because store owners and security firms really want to put away shoplifters, they claimed that video proof of the shoplifting was routinely deleted when there was no policy describing a deletion policy within Apple or SIS, only for video proof that the individual was not Mr. Bah to later be found.

in this case, facial recognition tech was unreliable entirely because of the people running it.

The HN title is bad. The teen was not falsely accused by unreliable facial recognition technology.

I sort of skipped through the actual headline trying to figure out how AI was involved. It was. But only after several blunders by people.

That's the case with all "bad AI" stories. Remember, an AI can't make a mistake until we have true AI. Until then, there's always human error - either the operator, or the programmer.

In this case, it was obviously a mismatch between how the different operators treated the 'name' field - one operator put in an uncertain ID, and another one trusted the system to be correct.

These stories are IMPORTANT. They might help someone think twice before either using or blindly trusting these systems.

There's still a huge difference between "people put in the correct parameters and AI nevertheless got the wrong result" and "people put in incorrect parameters and AI isn't a miracle worker that can correct it for them."

You wouldn't title a story about a guy hitting his thumb with a hammer "Man attacked by unreliable hammer"

If the teen was falsely accused and the false accusations came from the facial recognition technology then the facial recognition technology was, in fact, unreliable.

The false accusations didn't come from the facial recognition technology. They came from believing the criminal using this falsely accused person's ID card which did not have a picture on it. Facial recognition tech correctly matched the face of the criminal to the other video of the criminal.

While technically true, the use of tech is still notable here because it was used to launder the mistakes of humans.

There is an incorrect tendency to ascribe intelligence to AI, which can make especially laypeople prone to assume that if an AI system makes a statement, that statement has a certain sort of validity to it that isn't necessarily justified.

Garbage in garbage out applies to AI just like it applies to any other software system, and society needs to develop more resilience to this fact.

AI had nothing to do with the mistake. The error was in database with poor data quality.

The same would have happened if a human recognized the kid and matched him to the (stolen) id card on file.

You didn't engage with the core of the argument, which is that the veneer of AI is used to launder the human mistake.

Increasing the distance between database input and output gets in the way of human heuristics for recognizing and fixing such mistakes. Yes, the same could have happened without AI, but the probabilities are different.

Exactly, it's just that "Teen accused of shoplifting due to unreliable data entry" doesn't get as many clicks. It's not like the database itself was unreliable, or the facial recognition was unreliable.

Doesn't that make it worse though?

Drum up false charges and they get automagically mapped to the face with nobody able to undo anything. They're now forever a criminal. Honestly, it sounds extremely dangerous in a field such as criminal investigations.

In this case, it's really the problems of names, not faces. The wrong name got attached to the charge, but the right face.

This is precisely why the "fear" or rather stigma around facial-recognition needs to go away. If it weren't there, it'd have been expected and granted that the government's job is to have valid, verified and correct biometric information on the citizen (because they're generally the ultimate/final custodian of people's identity), and the facial-recognition would have to match against that verified data, instead of some random video or fake ID.

So instead we have scattered silos of partial and probably incorrect data in various places. FBI has a bit, Police department in state X has one, State Y's DMV has something wrong because X years ago this person was a victim of identity fraud, etc etc. It's a giant mess and no wonder the police and various entities in this example failed utterly. They're ultimately all victims of a system that has no single source of truth, yet we're tasking them with using "tech" to solve crime. So instead of fixing the source of truth, we blame the tech.

The stigma is there, because if it was in one centralized database, the issue would be much more pronounced. It is a lot easier to destroy a company than a government.

The false accusations did not come from facial recognition technology.

> in this case, facial recognition tech was unreliable entirely because of the people running it.

Wrong.

In this case the facial recognition tech itself was arguably not unreliable. The human operators were the unreliable factor.

The recognition tech reliably tagged the impersonator as Ousmane exactly as it was instructed to do. The system worked exactly as intended. It is the human operator whose intention was wrong.

This has nothing to do with AI being unreliable and everything to do with the employees of this SIS company going "yep kid's black, he's the one who did it" without half a thought.

> The human operators were the unreliable factor.

Separating the two things (operator and technology) is merely a technicality, in reality they are not transparent to the general public and should be treated as parts of the same.

If AI™ only happens in a black box behind closed doors with people getting the opportunity to make inconvenient results disappear and that is basically the way it is trying to be established then no, it is not reliable.

The operator is part of this whole system, if it can't be used without the unreliable operator, the tech is not reliable.

> In this case the facial recognition tech itself was arguably not unreliable.

You could train a fake money detection AI with cat videos from youtube, it would probably do something with images of cats. However I hope no one would try to argue in front of a court that the resulting AI would be reliable at detecting fake money. In this case a stolen "do not use as id" slip was apparently good enough to serve as input validation, I would be surprised if their database wasn't overflowing with bad data.

It wasn't validation, it was labelling.

The AI correctly identified that the faces of people in two videos were the same person. It did it's job. The person responsible for correctly identifying the person in both these videos failed to do their job correctly.

IMO, the operators of the AI are part of the production execution of that AI. There were not proper constraints in place to prevent this from happening, and it wasn't fixed after the first time(s) that it occurred.

Sure you can argue that the pure tech worked as designed, but the system includes the human element as much as the tech element.

The operators made it worse of course, whether enabled by AI or not. We've been dealing with this exact same issue, without AI, for decades when it comes to identity theft.

These "operators" that you say are part of the production execution of "AI" are also part of every single institution that led up to this person being here and existing. Everything from registering their birth, capturing their DMV details, capturing their details when enrolling at a school and opening a bank account, taking their photo for the driver's license, etc. All gooey, imperfect, corruptible and fallible people that taint the "chain of custody" about the person's identity.

>This has nothing to do with AI being unreliable

This has everything to do with AI being unreliable. AI is unreliable. It was unreliable yet again here. Anyone relying on AI as the evidence in a prosecution needs to have their faces rubbed in the ignorance. AI is as unreliable as any human being. In fact, it is more unreliable because people understand the limits of the reliability of humans and look for corroboration and examine the evidence with a mind on its potential shortcomings. Yet there is still this pervasive and pernicious belief that "Computers don't lie" which is completely and utterly false and worth repeating often, with emphasis until it is part of human collective understanding of the world to the same degree that water is wet.

"Computer says x" by itself counts for nothing. Anyone presenting evidence like that by alone should be presenting a huge red flag that something nasty is going on, via either malice or sheer incompetence.

The AI was reliable. It correctly identified that the individuals in two videos were the same person.

The problem was that one human recorded a name that they should have known was not real, and another person read that name in a report and arrested someone with that name. The story would have been no different if a human had correctly recognized the same person in both videos.

You seem right to say it is not a case of AI falsely accusing someone based on pure randomness, but that is no reason to dismiss what happened here. The most troubling aspects are that SIS added what amounts to false information to its databases, fabricated multiple false cases against Bah, claimed to have evidence it did not have, used it to detain him and hand him over to the Paramus police for arrest (who failed in their own due diligence, partly on account of this false information), and then attempted to hide what it had done, including, apparently, destroying evidence - evidence against itself, not Bah.

If a DNA test identified the wrong suspect because someone mixed up the names when entering them into a database I wouldn't say that there was "unreliable DNA tech".

Maybe other people didn't read the headline as I did. But if I hadn't bothered to click on it I 100% was just going to assume it was because the tech itself was unreliable and they picked out someone who looked like him.

Perhaps if I wasn't read up on the debate around this I wouldn't have thought that - but I imagine most people would assume this with confidence if they saw the headline on their feed.

You are making a valid, but relatively small, point about how the article is written. People who think they have learned something important from a headline without verifying what they think they have learned by applying critical analysis to the article are likely to be misinformed in all sorts of ways regardless of what they do read.

Note that we are both talking about other people - we, of course, are not like that!

Not just another kid, but a friend. Ousmane Bah is an honors student, while Mamadou Barrie is potentially responsible for a string of thefts and the impersonation of his friend to the police.

> "The arresting officer was able to identify the impostor as Mamadou Barrie, a friend of the Plaintiff, who apparently stole the learner’s permit from the Plaintiff," the complaint says. "These arrests specifically [noted] that Barrie had pretended to be Ousmane Bah."

> The detective subsequently submitted an information request "to the NYPD’s Facial Identification Section (FIS), which identified the photograph as potentially depicting two people, one of whom was purportedly Ousmane Bah – and the other was the actual thief, Mamadou Barrie."

That's why you pick your "friends" carefully.

The headline is accurate because that's one subject of the lawsuit. If I'm understanding it correctly after skimming through the filing(read especially page 6 and 9):

1. the plaintiff argues that in Boston case (page 6) apple store only took less than 10 minutes to identify the shoplifter, so they probably used facial recognition tech. Note the shoplifter left the store, apple found the incident, report it to police within 10 minutes, without actually detain the person.

2. I'm guessing the facial recognition is actually accurate because it was based on their previous security video, not plaintiff's real id photo(which apple probably don't have access all the time). So the facial recognition just linked one person with previous security video properly, and the id of person was based on previous record made by employee.

3. Thus there is nothing wrong here with facial recognition, but the title will definitely evoke fear and people's emotion since most people will not see the details.

The AI falsely accused someone because the AI was fed garbage data. If you’re going to build an algorithm that can levy such serious charges against people it certainly seems negligent to create a process that allows random unverified training data to be input into the system.

There are real people who suffer real consequences because of the programs we write. When things go wrong we don’t get to point the finger at the algorithm and the users and take no blame as the developers.

Reply on news.ycombinator.com