Comment by bad_alloc | Hacker News Reader

Comment by bad_alloc | Hacker News Reader

Parent Full thread

Decency and foresight is highly subjective. Actually encoding what we understand as "decent human behaviour" into AI is a huge problem: If you create a system that decides about giving somebody a loan in the US based on a financial data set it might come up with any of these hypotheses:

* Don't give loans to people living in [poor area].

* Avoid people with names that aren't similar to the most common ones in the database (i.e. foreign ones)

* When linking the customer data to their social media and you see the picture is dissimilar to [preferred ethnicity], do not give a loan.

Now without any malice from the developer, the system has become racist: It saw the correlation that in the US blacks and hispanics live in poverty more often that others [1]. It knows that poor people pay back loans less frequently and makes the rational decision to not give a loan to that group. This of course reinforces the problem.

But how are we to solve this? Introduce an additional column "race" to the data and bias the results with it? Would that not be just as racist? How do we give the system an awareness not to discriminate against ethnic groups, if the data contains implicit clues? This comes down to giving such an AI human intuition about such questions.

[1] http://www.ssc.wisc.edu/irpweb/faqs/faq3/Figure1.png

I'm not saying teach people some universal moral code, or suggesting we can accurately encode a set of appropriate morals into an AI. Just that people stop and think if what they are doing is appropriate.

It seems silly to take problems which humans cannot solve with all the data, intuition and humanity that we have, and then use that data to train machines to make decisions, we know will be flawed.

>Now without any malice from the developer, the system has become racist: It saw the correlation that in the US blacks and hispanics live in poverty more often that others [1].

If the data really suggests that these factors (poor areas, foreign names, social media) affect liklihood of default, should they be completely ignored? I'm sure credit scores and salaries are correlated to race in the US too. Should those be ignored? Just give out loans indiscriminately?

They're not ignored now by loan approval. Worse, inaccurate stereotypes are used as a proxy for some of these factors that may be relevant to a borrowers ability to pay. We don't live in a perfect world where people ignore certain factors.

Are you familiar with correlation vs causation? Without a causal link, making an inference based on correlation is unsound. Encoding unsound reasoning in an AI model, particularly when it reinforces an existing social imbalance, would be less than ethical.

"Weapons of Math Destruction" by Cathy O'Neil delves into this in more depth. It's a very valid concern, particularly in the way non-technical people are trained over time to give deference to the algorithm.

I'm not sure what you mean. AI doesn't involve "encoding unsound reasoning" into anything. That's more akin to how humans think and act, by making decisions based on heuristics learned from a lifetime of experience and societal influences. It would be unethical to encode a "race" factor to be considered in an algorithm, although even if it were included I think it would prove insignificant since race has no direct causation to ability to prepay when you consider all the other factors (e.g. poor white person in same neighborhood would have same default prob). However other factors might, such as the ones you have listed (where they live, credit score, etc)

I am familiar with the book you mentioned.

You still didn't answer my question. If credit score and income are correlated to race in America, should they be ignored from loan applications? What are valid factors to consider?

> If credit score and income are correlated to race in America, should they be ignored from loan applications?

Is correlation causation? Is it fair to encode a judgement based on correlation but not causation?

Or to make your position more clear, do you support racial profiling? If not, why not, and how do you justify that position, and how does that justification apply to judgements based on correlation but not causation?

(My position is that making decisions based on correlation but not causation is the general case for which racial profiling is a specific case.)

I already answered that. No race should not be a factor as it does not convey any info not explained by other factors.

What are factors that are causation? Presumably not where you're living and other considerations you determined to be implicitly racist

+1 for Weapons of Math Destruction.

You know, if you code your machine right, and give it the real information it wants (income level), an inference machine will detect that race is a worst proxy than the direct value and will not become racist.

It's when you start to deny proper information into decision making that those bad proxies start to look good.

But a correlation between income and race might be detected and used by the system as well, making it at least partially racist. Maybe no better than humans today but we should at least aim to make our AI not overtly racist.

Reply on news.ycombinator.com