‘Simple’ AI can anticipate bank managers’ loan decisions to over 95% accuracy
unite.ai
unite.ai
The paper is about the manual process of re-assigning a credit score on a scale of 1 to 15 based on other customer criteria. Really the fact that this process exists at all shows that their initial credit scoring approach is flawed or too simplistic. The argument of "just replace it with an if statement" does not hold up in this scenario.
So this is not a "if number big lend. If number small no lend" problem. Its a 15 way multi-class classification problem. They even give a baseline for what happens if they randomly pick or always pick the biggest class in the paper
> As is typical in machine learning we also report the Accuracy p-value computed from a one-sided test (Kuhn et al., 2008) which compares the prediction accuracy to the "no information rate", which is the largest class percentage in the data (23.85%).
So yeah, 95% is somewhat better than 23.85%.
I agree with the general sentiment that is is likely a fairly straight forward problem to predict if you are familiar with the bank's operating procedures as there is no way these individuals are making their own risk models and independent decisions. They are there to follow the rules and provide human accountability.
An error analysis on the items the model couldn't predict would definitely have been most interesting.
Humans are both biased and with high variance (not to mention corruptible), but the algorithm can benefit from much better scrutiny and ensure uniform application of the criteria. If a human overrides, then they got to have a good reason.
[1] https://ec.europa.eu/info/law/law-topic/data-protection/refo...
No bank in any credible jurisdiction will have an automated system approving 1bn USD equivalent loans any time soon. A typical system would be loans up to a certain amount can get approved automatically, up to $xM by a credit officer, up to $yM by a senior credit officer, anything over that by the credit committee. Regulators push back very hard on automated decisionmaking for large loans particularly because of "default correllation skew"[1] problems as were revealed in the 2008 crisis. Relatively few bad decisions on big loans can push a bank into difficulties if they are not well-capitalized. This is particularly a problem for automated decision-making because as loans get larger they also get more idiosyncratic and therefore it's much harder to fit a model with confidence because there simply aren't enough data points.[2]
[1] Often credit quality for a group of loans rises because of idiosyncratic factors but deteriorates together, so as loans become more risky in an adverse economic environment the correllation of the default probability between the loans goes up. An intuitive way of thinking about this is in housing loans. If 3 or 4 of your neighbours default on their loan, property prices on your street will go down (because the banks will be trying to sell all those houses at once), making it much more likely you and the rest of the residents will default too.
[2] Say I'm trying to approve a 200k loan to expand a pizza restaurant. If I'm in a big bank I have hundreds of similar loans to use as data points for pricing and risk. If I'm trying to approve a 200M loan to build a luxury hotel complex that includes 5 restaurants, accommodation, retail etc it is completely a one-off. Even if I'm the largest commercial lender this loan will be unique in my portfolio. I will have many other large loans but there will be lots of idiosyncratic factors that make them different.
In the case of a loan, if the credit score is high they'll probably qualify. If it's low, they won't. All other variables are very minor in comparison to this one, which already encapsulates almost all of the relevant information. This is by far the most common situation for practical machine learning.
Else they would not be able to secure funding for the idea from upper management because they had learned that 'data is the new oil'.
The second VS part always needed/needs to be how this project enables the product to be (or become) a platform where the company can the run value added services on top (value adding for the company, not the customer).
Imagine a car being a platform and you can book additional horse power for the upcoming trip. Or change the background image for the instrument panel. Or pay extra to natively have access to your Spotify playlists. And then datascience the user data to recommend even more stuff resulting in $$$ (by imagination from management and fuelled by slide decks created by some junior strategists that have never even owned a car but presented by the client account lead in a lush retreat).
And in private non business customers there are even a significant amount that have less than nothing to their name, living of welfare and still don't see, why they should not receive a loan from the bank for the newest iPhone. Or a new car. Or TV because "football's coming home".
Excuse my snarkiness, but the stories she told me from her apprenticeship in the bank, when she was customer facing were not painting a good picture of humans (Btw. well off or not). But that is probably the case in any consumer facing job.
I am not siding with banks in general here. I was just exemplifying one aspect to show how it is not always black and white.amd how there can be unreasonable customers.
[Edit typos]
Also, the burden on the bank for an overdraft is near zero, so why is it $35 for the customer? Perhaps a fee of only $5 would be less regressive?
Anyhow that’s really awful and needs to stop, but I don’t see eliminating overdraft fees making a real dent in the advantages wealthy borrowers have.
How could underwriting be changed to identify truly creditworthy borrowers who don’t have any financial or real assets? We already know that simply lowering underwriting standards has problems.
[1] Their words, not mine.
Excuse my snarkiness but as someone who's been poor and needed a phone, I'd have got a loan to buy a 2nd hand phone if anyone would've given me one. Trouble is, both the companies providing the phones, and those providing the loans seem to prefer that I get the brand new one.
I myself had to live on 345 Euros a month for quite some time. And even longer was living below what Germany considers the amount of monthly income that one would qualify to not be called poor ("Armutsgrenze"). I know the feeling and hear you. The only difference is that I could not even get a contract as no telco would find me creditworthy enough. Needed to do prepaid and see how I would get any old mobile.
> The real benefit of any form of machine learning is detecting functional relationships between variables (ie, "when this AND NOT that OR this"). It's just that these relationships don't provide a real benefit in 99/100 real world use cases.
And the real problem with humans, ironically enough, is that due to narrative fallacy, confirmation bias, etc., humans vastly overvalue the contributions of these special circumstances.
If a dumb 2--3 variable rule predicts your friend will go to the movies in the weekend, you are likely to go, "Yeah, maybe. But she complained of that headache this morning, maybe there's something deeper there that also makes them not want to go to the movies."
In most cases, you'll be wrong to override the dumb rule.
So why do people do this? Well, the original movie-going prediction might only have been a 55 % shot. So 45 % of the time, your friend won't go to the movies, headache or not. But if they don't, you'll think you made the right call considering that headache.
If they do go, you'll end up thinking, "Right, of course. They had the headache but their friend really wanted to go so of course they would endure it."
In other words, you'll think of a way to frame your incorrect call as the right one. (Hindsight bias.)
Do the AI's 5% discrepency picks perform better or worse than the human's picks?
The title is worded to suggest the AI can, or is very close, do the human's job, which could totally be so.
But it could also be that the AI loses 5% vs the human, and the bank only makes 5% on loans in the first place, and so losing 5% of them is like totally erasing the entire income of the bank (from loans) which makes the AI a complete Hindenberg, rather the opposite of the implication from the title.
Without more data it's not clear if the algorithm is actually effective - e.g. if the approval rate of these loans is 80% for instance, a formula of "always return true" could be said to be '80% accurate compared to human decisions'.
And if the AI makes the same decisions as a human 95% of the time, but then makes horrific errors the other 5% of the time, it wouldn't be an appropriate replacement for the human. This is the same issue with self driving - it doesn't matter if you make great decisions 99% of the time if 1% of the time you decide to drive into a wall (until you have got your error down to a place where it's safer on aggregate than a human).
That way, you can compare the more fancy stuff to something to see whether you're really improving.
You also get to evaluate the economic gains of the more sophisticated model against the development, maintenance, and data costs of it, compared to the dumb one.
(Another good baseline is returning a random historic result in proportion to how often it occurs. It sometimes helps against exploit attempts at the expense of data.)
Here, those 5% of cases are likely business-ending or business-making: precisely because they arent naively routine.
Also worth noting that the arithmetic expectation is only a good way to measure the profitability if we are talking about small amounts compared to your total wealth. For any other case, you should use the geometric expectation of total wealth to evaluate options. (This is equivalent to maximising log wealth, the Kelly criterion.)
The problem in this case was defined as reproducing the human scores. Mainly, it's a demonstration of the information content in the data and the scores. To me, it demonstrates how simplistic the human scores are.
With information about historic outcomes, we could have compared the effectiveness of the "credit score method" with some other AI algorithm that was optimizing total value.
When you start taking larger bets, you want the geometric expectation of total wealth, i.e. (current wealth + outcome)^probability.
(This is the Kelly criterion for judging significant opportunities.)
I even worked on a loan program years back to centralize a bunch of acquired banks. They literally told their staff not to tell customers there was nothing they could do and a set of rules in the computer made 99% of the decision for the loans now.
I wouldn’t doubt that if you sat down, and created a data set/data collection scheme to gather this data you could make an algorithm to closely mirror the outcomes of a loan decision. However as a human, the loan officer might simply add a new criteria as desired.
There will be elements of AI which are useful, but ultimately banks will want to know why a certain decision was made, and want to incorporate their own economic calculations and forecasts into the model.
The benefit of using SQL rules engine in a financial setting is that you can prove causality and intent throughout. Why a customer was declined for a loan can always be traced deterministically to some rule SQL that legal previously approved per regulations in that jurisdiction.
Toward Data Science has a neat example [1], scroll to the end to see the sample "scorecard" which can be implemented as lookup tables and ifs. Example shows how the customer gets certain points depending on the age group, home ownership and income group. Sum up these points and you get the total score, which then translates to probability of getting your money back from that customer.
[1] https://towardsdatascience.com/intro-to-credit-scorecard-9af...
I understand startups need funding from VCs whose idea of technology is "BLOCKCHAIN AI" but, being interested in AI, the amount of spam I have to dig through to find actual AI work is insane.
What's left is a bit of judgement as to whether an applicant is temporarily showing wrong data, eg if they're a student starting their first job, what is their income, low or high? But basically it means the bulk of cases can be done by machine and the loan officer looks at some corner cases.
Note this person is also an actual person, i.e. capable of being held to account. I once worked with a prime broker who decided to waive the red flagged checks on a certain client in order to win the business. The guy blew up badly, the bank lost money, and he lost his job.
Even if an AI can produce seemingly the same results as a human, it should be out of the question anyway to let an inscrutable black box make decisions over people's lives. Because at least with a human you (their boss, or a judge, etc) can ask them "Why did you decide that?" and they can tell you. A racist or mysoginist or religious human can be identified and fired or corrected etc. How do you judge if an AI is giving inhumane decisions?
The decisions themselves can't really be judged, only the process that generated them, and you can't see that process in an AI.
If someone doesn't get a loan, and someone else does, you can't tell that wasn't right just from the final result.
Even if the results "look" wrong, like only 30% of black people get the loan while they made up 40% of applicants, even that could possibly be exactly correct, but you can't know if you can't see the process.
But a human can be asked, and simple algorithm code can be read.
Probably these days the human has so little discretion anyway that the corporate policy is the algorithm and the human is pointless anyway except as a sham human-looking interface to appease customers. It helps sales to have a human, but the human in fact wields none of the human power that the customer wants a human for.
If the number of variables is low, then the extant bias may be cooked into the inputs and the AI result is then inevitable. For example, income will be suppressed all other variables being the same, that's an immediate loss in approval chances.
If crime maps and the like go into some risk profile on the safety of the property, well guess what, that'll be effectively racist and/or lower loan preapproval.
It doesn't have to just be that the race checkmark/dropdown becomes an input.
But you're right, the AI is a black box. Rules engines or other calculations at least can be back-traced, and maybe counter-weighted.
A great deal of structural and persistent racism comes down to housing: where you live. It doesn't take a die-hard Zillow user to know that where you live will signal things to the loan application algorithms, regardless of "race blind" applications if that is even a thing.
Thus, higher loan rate, lower maximum value, less access to various stratifications of neighborhood "eliteness". And the transaction rate/speed on housing is really low (annoyance to move, realtor fees, closing costs, PITA to shop for new/sell your old), so policy to address the bias would take a longer amount of time to actually show up in the stats than its political survival time (because so many well monied interests will lobby for its removal).
And thus, the more things change, the more it just stays the same.
So a loan can be good or bad. The loan officer can rate it as good or bad. And the software can rate it as good or bad.
To evaluate the situation, would like more than the "95%":
(1) Would like to see the arithmetic that yielded the "95%". (2) Would like to know the rate (probability) of false positives, when the software said the loan was good but it wasn't. (3) Would like to know the rate (probability) of the false negatives when the software said the loan was bad but it was good.
And, really, when the software and the loan officer disagreed, what were the ratings of the applications and when was the software correct and when, the officer correct.
Finally, what was the average cost per mistake for the false positives and for the false negatives. E.g., a false negative could cost the bank some business but a false positive could cost them $millions.
The author looks at empirical evidence (experiments) and shows that you can take experts' predictions, use a super-dumb statistical model of what they do, and then outperform the experts' judgments with a simple linear model!
- Performance against a fair dice - Performance against a group of humans trained to predict bank managers' loan decisions
From the paper:
> As is typical in machine learning we also report the Accuracy p-value computed from a one-sided test (Kuhn et al., 2008) which compares the prediction accuracy to the "no information rate", which is the largest class percentage in the data (23.85%).
So fair dice 23.85%, model 95%.
That said I bet a human who had read the banking rules and regulations and recommendations on lending could easily match this performance.
If we sweep models f1,...fn across the validation set ... and choose max() of scores() of f1..fn on V, we get f*.
Now if your issue is that f* might be an "unlucky draw", you're right. But there is no statistical way of fixing this -- if we know (via experiment, etc.) what the true distribution is, we can meaasure |f* - t| -- but if we knew this, we wouldnt bother finding f*.
If you want a mechanism to mitigate these problems, there is one main one: the scientific method. To test whether f* poorly reflects t, go and do some experiemnts. If you can't, then you wont know.
(Hence: there is no way of doing science via "mere statistics". It is the experimental conditions constructed by concept-laden, in-the-world, experimenters which are able to obtain the sequence of datasets needed to give confidence to any given model. ML is therefore not able to know anything, its "conclusions" enterily derivative-of, and limited-by, human experimentation. The intelligence occurs in the experimental design, when that's done, everything else is "stamp collecting").
For large N sizes the models will tend to converge on the same logic. It’s actually much more of a problem with small samples. Simpler models like linear regressions and decision trees could even be deterministic.
if loan_applicant.credit_score > minimum_credit_score
decision = "approve"
decision = "reject"
1/3rd of salary is more than enough to buy you a home. That's enough to get a 200k loan for 30 years while making average salary (1700 net euros).