Hacker News Reader

> The Apple Card bias is a real problem. Who cares whether it's about gender or not? If it's not about gender, the responsibility lies on the bad actor to prove otherwise. The most logical conclusion - based on the events that occurred and the info the general public has - is that it's about gender.

And I work with credit risk models and can tell you there is a negligible chance that gender is causing a 20x credit limit increase. No protected variable will have so much influence, and Apple/GS are not even legally allowed to directly reference such variables.

It may be the most logical conclusion for the general public, but all they see is the output of a black box. DHH jumped to conclusions that it had to be something about gender, with n=1.

Who says there is a problem that needs fixing? Demanding that women and men always receive equal limits is not about fairness, it is a very radical notion that makes discriminative credit risk scoring almost impossible.

I am not saying that Apple/GS did not mess up here, and could have been more transparent, but, in their defense: Oftentimes you can't be more transparent (like when Google closes an account for posting something illegal, they have to keep mum, while the user loudly complains online that they got banned for doing nothing), an help desk person is never going to be able to give an explanation for a model decision (this is usually a good thing), and DHH seemed to be barking up the wrong tree all together (what if the different limits were the result of a, completely legal, influencer program or randomized trial?).

It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans.

Similarly, as this is an ongoing investigation now, and they probably anticipated that, Google was probably legally very limited in what they could disclose. I bet some people with all the knowledge of the data - and colleague snooping are shaking their heads at some of the public perception of this case as union-busting.

The opposite of accepting a coincidence once in a while is viewing everything through the lens of intentional discrimination.

> It really does matter how it got that way. If 20% of people of color vs. white people have a low income (a non-protected variable), I am perfectly allowed to use that variable to deny them a loan more often, even if the result is that 20% of people of color don't receive a loan. That's not a racist algorithm, however it would be racist if I start ignoring income of certain people as to achieve racial demographic parity in loans.

It's not that simple. The principle of "disparate impact" means that you can't, in general, make the decision in a way that adversely impacts people of color just because you didn't directly consider race in your decision.

You can use income in your decision because considering income "is necessary to achieve one or more substantial, legitimate, non-discriminatory interests." Income is directly related to ability to repay a loan, so you need to consider it. You can't just use any old variable you want to. You can't deny people for a loan based on what genre of music they listen to, because that has no justifiable connection to whether they will repay a loan.

Everything, including income, is eventually correlated with race. The point I was making counters "does not matter how, just the outcome matters": The model disapproved you not because you are black, but because you have a low income. There is societal racism there that needs addressing with policy and regulation, not by handicapping your model by throwing away non-protected variables. The -perfectly legal-outcome will be that fewer people of color receive a loan.

I know that you can't use just any old variable, but I tried enough to know that music genre would probably be an informative feature (dibs on providing loans to classical - and Judeo-Christian religious music lovers, you are free to underwrite the dubstep - and ghetto rap fans).

Music preferences undoubtedly would be correlated with risk. You still can't use them because of disparate impact.

To get away with making a decision in a way that has disparate impact, you need to have a legitimate need to be making the decision that way. In the case of income, you can justify needing to use it in your decision, because income is directly connected to ability to repay. In the case of music preferences, you can't, because there is no direct connection to loan risk.

> If it's not about gender, the responsibility lies on the bad actor to prove otherwise.

How do we know they are bad actor if we don't yet know what really happened? Imposing guilt on someone until they prove themselves innocent is a bad way to run a society.

It is not even close to reasonable to conclude that the Apple Card thing had anything to do with gender. There is no reason to believe that the end result was because of this. Especially given all of the reports of women getting higher limits than their male spouse. This is almost as logical as concluding that vaccines cause autism.

Why is gender discrimination equivalent to autism and vaccines here? There's lots of standing evidence for gender/race discrimination in finance and zero reliable evidence to support vaccines causing autism. You can come up with a better equivalence than that...

People assumed gender discrimination because you had a couple with good credit scores and the woman was discriminated against by the algorithm. The only data we have indicates that it was gender. Of course there could be some other reason, but the obvious conclusion we have based on the information we've got is gender.

My wife has a 3x higher limit on her Apple Card and better credit than I. Gender has nothing to do with it, it's just a new hate on Apple train.

It’s ridiculous because there is no evidence, just misinformed anecdotes. There is no field for gender on the application. The “equivalent” factors cited (income, credit score) are all irrelevant for credit limit decisions, which are primarily based on previous revolving credit usage. Given that DHH owns basecamp as an LLC, and not a corporation - known as a disregard entity for tax purposes - means he is likely putting all business expenses on revolving credit cards, which would create a massive disparity. That’s just one of many plausible explanations. DHH didn’t even understand that he is not in fact in a community property state - although this wouldn’t have been relevant either. And there are many counter-examples that our confirmation bias doesn’t seem to be interested in:

https://mobile.twitter.com/jimcopus/status/11937337466141286...

Furthermore, Goldman issued a public statement confirming that they reviewed the complaints and they are not using gender as a parameter. This would be trivial for an investigator to confirm, so it’s unlikely that they’d be stupid enough to outright lie about it.

Jumping to the worst possible conclusion based on a twitter anecdote is not helpful to anybody.