Comment by heavyset_go | Hacker News Reader

Comment by heavyset_go | Hacker News Reader

This article is a prime example of law enforcement passing the buck of responsibility for their own actions to a black box system by claiming that the system's output is "unbiased".

> “You may wonder why you were enrolled in this program,” the letter continues. “You were selected as a result of an evaluation of your recent criminal behavior using an unbiased, evidence-based risk assessment designed to identify prolific offenders in our community. As a result of this designation, we will go to great efforts to encourage change in your life through enhanced support and increased accountability.”

> Last year, a Tampa Bay Times investigation revealed that the Sheriff’s Office creates lists of people it considers likely to break the law based on criminal histories, social networks and other unspecified intelligence. The agency sends deputies to their homes repeatedly, often without a search warrant or probable cause for an arrest.

Now law enforcement gets to absolve themselves of accusations of bias or targetting because they assume that their computers can't be "biased", even though they only buy systems that confirm their own hunches about which people are criminals or not.

If anyone complains, they can say they were only doing what the supposedly unbiased system told them to do, after all.

Not to mention,

"The Times found being named a Sheriff’s Office target could have serious consequences. Deputies showed up at homes at all hours of the day and night, writing tickets for violations like overgrown grass and making arrests for any reason they could find. By 2020, some 1,000 people had been ensnared. About 100 were 18 years old or younger."

"Since we have determined that you are a criminal, we will ensure that you are a criminal."

> writing tickets for violations like overgrown grass

If I didn't know better I would think this story is from The Onion.

This is precisely why I voted against abolishing cash bail in California last year. Because they were going to replace it with some "algorithm." I hate cash bail, I think it's garbage, but this black box approach is worse.

Algorithms can be okay, as long as they are transparent. The problem with using things like AI is that the parameters or the algorithm are often hidden from the harmed individuals. And the law enforcement community can kick the can down the road by saying "but the software says...".

Except this is a Florida county.

This is exactly why one of the lesser known provisions of GDPR (https://gdpr-info.eu/art-22-gdpr/) addresses "automated individual decision making" (although there are likely exceptions for government activity, but they may require a law).

It’s hilarious because the shit they’re basing their model on (arrests/contacts with police) is biased as hell

Even if you haven't had any contact with the police, they can still put you on the list if they suspect you committed a crime. The manual they follow[0] uses a point-based system, and "suspicions of an offense" counts for half the points of actually committing the offense:

> • Half of the applicable point values are awarded for suspicions of an offense listed in 1-4.

[0] https://s3.documentcloud.org/documents/20412738/ilp_manual01...

Looking at page 13, they talk about their harassment list for at-risk youth, and mention these as 'identifiable risk factors':

- Hanging around in public

- Being a victim of personal crime

- Coming from broken homes

Page 72 show that they use records of children who experience physical abuse, sexual abuse, neglect, or witness domestic violence to put them on lists. Page 73 shows the "At-Risk Youth Spreadsheet" which is both terrifying and strikingly banal.

… and well known.

For example, we have known that broken homes lead to all sorts of problems a much greater proportion of the time for ages. And one of the things that becomes more likely is crime.

Not that that makes a computer algorithm’s predictions true. Just saying that it’s not terrifying to point out that there really are such things that increase risk that we are well aware of.

This in my opinion is a point that brings a lot of well-needed nuance to this issue. All or at least many of these facts or assumptions are all relatively reasonable or well-studied, that past criminal records and traumatic upbringings can lead to crime, but when put together under the dubious question of is the person a criminal can be greater than the sum of their parts. A lot of people come from broken homes or experience trauma and never commit a(nother) crime. The fact that these assumptions are standard practice has only legitimized taking a black-box algorithm at its word. It's not all just bias, but this kind of system certainly can incentivize it when making seemingly authoritative predictions.

The nuance doesn't matter, though. If you tell a cop that person A is 1% likely to be guilty of a crime, and person B is 3% likely to be guilty of a crime, they are going to focus their efforts on person B - despite both people being more likely than not, to be innocent.

When being harassed by police is itself sufficient to turn a person into a criminal, or ruin their life, you end up in a situation where everyone is doing what is 'reasonable', but leading to outcomes that are both unfair, and even negative overall, to society.

> When being harassed by police is itself sufficient to turn a person into a criminal, or ruin their life, you end up in a situation where everyone is doing what is 'reasonable', but leading to outcomes that are both unfair, and even negative overall, to society.

I agree. I just think that when critics read articles like the one in this post, they might walk away with the idea that all of these assumptions are fine and that critics of these systems are being alarmist. They might see these metrics as reasonable considering they reflect trends in actual criminals, but not understand that they are being used to predict criminal behavior, which like you mentioned with the two almost definitely innocent people is not the same thing.

>Just saying that it’s not terrifying

If you think China's social credit system isn't terrifying, then I guess I wouldn't be surprised if you didn't find this terrifying.

I said it’s not terrifying to point those facts out. I do think trying to use them in a prediction system is terrifying.

This article about the program[1] directly addresses your assertion, specifically by one of the professors whose research was used to justify the program:

> The Sheriff’s Office says its program is based on research. It points to a 2015 study that found young people who had experienced multiple childhood traumas were at a higher risk of becoming serious, violent criminals than those who hadn’t.

> But David Kennedy, a renowned criminologist and professor at the John Jay College of Criminal Justice whose research is referenced in Pasco’s manual, said the associations between childhood trauma and criminal behavior are “extremely weak.” He said using them to make predictions about individuals “flies in the face of the science.”

> The methodology used by the Sheriff’s Office, he added, was likely to generate a large pool of children, the vast majority of whom would never get in serious trouble.

> “There’s nothing — absolutely nothing — that can be fed into even the most sophisticated algorithm or risk-assessment tool based on information available when someone is a child that can say this person is going to be a criminal later on, much less a serious prolific criminal,” he said.

[1] https://projects.tampabay.com/projects/2020/investigations/p...

I think making predictions from these things is absurd (and terrifying).

Reply on news.ycombinator.com