Hacker News Reader

Comment by alphabettsy | Hacker News Reader

Some commenters here mentioned they may have setup the test incorrectly and that may be, but I think the problem this highlights most is that technology used improperly, especially by law enforcement can have major ramifications and consequences. Is the contractor that makes software for your local department going to follow best practices and have the algorithm audited by experts? Will they release the code? Those are real considerations for any program that has the potential to help ruin someone’s life.

It also possibly highlights the fact that algorithms are not immune to bias when they are designed by humans. Obviously I don’t think this bias is intentional, but so much of it isn’t and happens anyway.

View on HN

I think a fun thought exercise is finding the fine line between tech and guns/alcohol/cars. You cannot sue a gun/car/alcohol manufacturer if their product is used to injure someone because it functioned as designed but was used maliciously. How does that legal precedent work when extrapolated to tech and something like facial recognition? If it worked exactly as designed and we know it has a margin of error (or can be used improperly and have disastrous results, like a car or gun), could Amazon or a tech administering it be liable for someone falsely imprisoned?

> could Amazon or a tech administering it be liable for someone falsely imprisoned?

I would wager "absolutely not" for the exact same issue as firearms or cars. The person who mis-applied the technology or a middle man vendor though? Unless they fall under qualified immunity, there's your fall guy.

It's still ethically questionable. In fact I'm struggling to come up with a better example than facial recognition tech (except other mass surveillance). Maybe cutting corners while developing driverless cars that results in the death of a pedestrian.

Almost every engineering discipline has a code of ethics [0][1][2][3]. It's time software "engineering" grew up and did the same.

I rarely see ethics mentioned on HN, and granted, people's view differ. But it's weird we're not having that conversation at all.

[0] https://www.raeng.org.uk/policy/engineering-ethics/ethics

[1] https://www.ieee.org/about/corporate/governance/p7-8.html

[2] https://www.nspe.org/resources/ethics/code-ethics

[3] http://www.asce.org/code-of-ethics/

You're right! This is a critically important conversation that we absolutely need to have within our profession. It's very often ignored and there's no support system for people who take ethical stands.

So. Let's talk about ethics. I, personally, subscribe to the ACM code of ethics.

I think Rekognition, as built and presented, falls fully within that strict ethical code. It can be put to uses that are unethical, but that does not fall upon the people who made it. Certainly, an engineer creating a system such as the one the ACLU created would be acting unethically.

If firearms manufacturers produced guns that fired straight over ninety percent of the time, but no guarantee of bullet direction beyond that, they'd probably have a lot of lawsuits on their hands.

I think a better analogy would be a bridge or building- they are regulated by a rigorous code that will exonerate the engineers behind the implementation only if it was followed to the letter.

Will that slow things down and make them more expensive? Absolutely, but that’s the cost of safety.

In this case though what is probably most lacking is public knowledge about the shortcomings of such systems. The public needs to understand that 90% accuracy, while it sounds high, mean it’s wrong a lot, and the chances of it being wrong if it was continuously run all the time are actually quite high.

Would a camera manufacturer be liable if the camera's image processor happened to cause one person's photograph to look like another, which then caused a human looking at that photograph to misidentify someone?

Malice is beside the point here is the it? The concern isn't an evil contractor, so much as an incompetent one.

Car manufacturers are sued for incompetent use of their products.

Reply on news.ycombinator.com