The Mythos of Model Interpretability
arxiv.org
arxiv.org
Simpler statistical models, and particularly linear models, provide easy-to-grasp "answers" to that question -- for example, "to get more of y, we need more of x1 and x2, and less of x3."
An entire generation of leaders and professionals has been trained to think this way by statistics professors and teachers who have been rigorously trained to think that way too. (For background, I highly recommend Leo Breiman's now-classic paper, "Statistical Modeling: The Two Cultures:" https://projecteuclid.org/download/pdf_1/euclid.ss/100921372... )
Alas, there's no going back. Simpler models do not work for complicated tasks like, say, predicting the next best move in a game of Go, for which there are no 'key variables' in the traditional sense.
Robin Hanson summarized this well in his article "Who about forecast accuracy?" [0]. Here's a nice quote:
> The benefits of creating and monitoring forecast accuracy might be lower than we expect if the function and role of forecasting is less important than we think, relative to the many functions and roles served by our pundits, academics, and managers.
> Consider first the many possible functions and roles of media pundits. Media consumers can be educated and entertained by clever, witty, but accessible commentary, and can coordinate to signal that they are smart and well-read by quoting and discussing the words of the same few focal pundits. Also, impressive pundits with prestigious credentials and clear “philosophical” positions can let readers and viewers gain by affiliation with such impressiveness, credentials, and positions. Being easier to understand and classify helps “hedgehogs” to serve many of these functions.
> Second, consider the many functions and roles of academics. Academics are primarily selected and rewarded for their impressive mastery and application of difficult academic tools and methods. Students, patrons, and media contacts can gain by affiliation with credentialed academic impressiveness. In forecasts, academic are rewarded much more for showing mastery of impressive tools than for accuracy.
> Finally, consider next the many functions and roles of managers, both public and private. By being personally impressive, and by being identified with attractive philosophical positions, leaders can inspire people to work for and affiliate with their organizations. Such support can be threatened by clear tracking of leader forecasts, if that questions leader impressiveness.
FWIW, having worked for a while in quant finance, I can testify that this precisely nails it. Even when it would be cheap, easy, obviously worthwhile, and likely impactful, projects to switch from politically-controlled, simplistic regressions couched in the buzzwords of academic finance and instead embrace principled statistical computing, proper model fitting hygiene, machine learning, etc., such improvements will never be considered. There's too much political gatekeeper rent-seeking at the higher levels of the management who oversee the investment process. Sure, they'll claim plausible deniability by saying something is "a black box" or making a fallacious comparison with an algorithmic trading firm that lost money, but the real reason is to protect their fiefdom, over which they are the lord and master because of their credentialed authority in some "interpretable" buzzword.
[0] < http://www.cato-unbound.org/2011/07/13/robin-hanson/who-care... >
I thought the exact same thing going into finance. If real money is on the line, surely they care about which techniques are most rigorously justified in a given model context, right?
Absolutely not. The incentive schemes from clients (who often don't actually fire bad investment managers when they should, and who also engage in chasing returns momentum despite the long and, by now, boring and uncontroversial history of that not paying off) don't actually punish inefficiency. It's a big political mess.
Really the best you can hope for is that they'll hire you for marketing purposes. Hey look, we brought in a shiny new expert in deep learning -- we're cutting edge, we swear! They won't actually let you do any real work with deep learning, of course. It will all be Excel jockey bullshit on factor models in which you'll do obviously fallacious things like directly compare the t-stats of two different model fits as a means for model selection. Maybe you'll write an ineffectual white paper on something slightly more advanced from time to time. But the big reason you're hired is to look good on paper and smoke cigars and drink brandy with the right person who wants you as a political darling in order to win arguments from authority about how you definitely should not migrate away from Excel/VBA.
This is not hyperbole, sadly.
This doesn't count sell side marketing work, or slow money (insurance, mutual fund) buy side work where people get paid for Assets Under Management versus a % of the gains.
Some firms also don't even bother to give any details whatsoever, and the job offer will simply say there is a discretionary bonus, no percentage, no description at all. I actually turned down a hedge fund job because of this. I told them that in order for me to feel comfortable accepting an entirely discretionary bonus, with no baseline or agreed upon way of relating it to base salary or profits and losses, I would need a much higher base salary, and they weren't willing to negotiate about it.
The number of firms, even among extremely quant-heavy hedge funds, who award bonuses in a manner that is not overwhelmingly political is exceedingly tiny.
What this means is that the same political incentives affect even most hedge funds, and so they care far less about the mathematical rigor of what they are doing than about how to sell political stories about it. If "interpretability" sells political stories, then that's what they'll do.
Where are the hedge funds using sophisticated statistics? Maybe Renaissance. Maybe. Where else? Certainly not DE Shaw. Certainly not PDT. Certainly not G-Research. Certainly not Coatue. And on and on. When you interview at these places, and see how the sausage is made, it is eye-opening and alarming to understand just how little their business utilizes or cares about mathematics, statistical rigor, and often not even proper software design. They are just more of the same kinds of shoddy software shops cranking out ad hoc code for rapidly varying political whims, but with far better branding.
I'm trying to get out...
It isn't just about having great models. 51% of the world can't beat the median return. Everyone has great models. It's figuring out where the models are wrong that matter. (Example 1: Most Mortgage models assumed that housing values in all US markets couldn't go negative at the same time) Sometimes that's by qualitative insights. But that's very very hard. And sometimes it's by having someone give some info that they shouldn't.
I think this is expressed in the paper's discussion section:
"We should be careful when giving up predictive power, that the desire for transparency is justified and isn’t simply a concession to institutional biases against new methods."
Prediction accuracy is not the only goal. There are only a handful of times in my life where I've seen a machine learning task where the training data was a perfect match for the business objective. The training objective is a proxy for the actual objective, where the gap between the two includes a lot of systematic bias. Interpretability is necessary to make sure the model isn't merely exploiting that gap. (And also to motivate further improvements.)
Instead, 'interpretability' is usually taken to mean something more akin to: which readily human-understandable cognitive structures (out of the set of such structures that I am the authortity on) is the model directly (and easily) mappable to? By mapping onto those structures, I can now weave a story about the model within the domain in which I am the established authority. If such mappings don't exist, or are very difficult to compute, then my authority is threatened.
Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective.
Given that when prediction accuracy is paramount to the definition of success you still see politics trumping science, I have very little confidence that when you move to domains where the objective criteria of the models differs from the real-life objective in non-trivial ways, then somehow 'interpretability' will be used appropriately rather than overwhelmingly just as a political tool.
That fact has nearly zero relevance to what I'm saying.
When I say that people look for mappings from the model to a space of cognitive structures over which they have authority, it could be (and observed in practice always is) a smattering of all the different things talked about in the paper -- authority over which diagnostics represent 'trust,' authority over which visualizations are adequately informative, authority over explaining 'how' an algorithm works, and so on.
Though all of these are discordant takes on the specifics of interpretability, as discussed in the paper, they still are unified in the sense that the true intention in appealing to any of them is to map things into a domain where someone wields the authority. It's not about actually caring about trusted diagnostics, or caring about quality visualizations, or caring about communicating the intuition behind a model. Those are just the plausible deniability excuses proffered to try to win the battle over which faction's authority gets center stage.
I don't require that anyone have a cross-domain consistent view of the specific definition of interpretability in order for my claims to be a viable explanation, and so it is perfectly compatible with the paper. The paper looks at the on-the-ground specifics used in the arguments over interpretability, meanwhile politics explains the behind-the-scenes motivations of any of those specific choices.
The Moral Mazes chapter Dexterity with Symbols is a good place if you want to read on the evidence about how this is carried out in management bureaucracy. It has actually been formally studied in some interesting ways.
What I see people mean by interpretability is one step back. It's literally just "what is the model actually doing?" in few enough concepts that meat machines can hold them in memory. Sometimes accompanied by a second question "is the thing it is doing dumb or smart?"
In my field (search) there are large classes of things that "work" but are dumb or annoying, or just not the product we want to provide, and we have to make product decisions to avoid them. So in that context figuring these things out is really important.
>Having said all of this, the most egregious abuses of the nefarious side of 'interpretability' that I have seen have all come in quant finance, which is a situation in which the optimization criteria are literally the identically same thing as the actual objective.
You know more about this than me, but it seems like there might be big differences between models in how they respond to unquantifiable downside risk of rare events. How do you deal with that without interpretability? How do you examine the assumptions the model is making that you may not want it to make?
Rather that try to define quality and think about the processes which could bring it about, they have a "I know it when I see it" attitude to quality. They value their personal discretion to promote, bury, editorialize and censor far higher than actually being a high-quality truly open forum.
Whenever I sit down to build a new model, or work with a new-to-me model, I start with questions like:
- What questions was this model built to answer? Be careful when asking other questions, and know the limitations of those answers.
- If it's been in use for a while, how effective has it been? How is that efficacy measured? A highly accurate decomposition of historical behavior may involve a lot of post-fact knowledge that has no predictive power, so be careful interpreting cross-use results.
- What are the ranges and distributions of the input variables? What are their boundary conditions? This is not always applicable to all variables, especially unstructured data, but when it is applicable, it is usually straightforward.
- What are the ranges of the model outputs? Do they have any boundary conditions? Do boundary effects impact the situations in which the model can be applied? For example, when working with a binned variable, often bins 0 and n include less homogenous data than the rest of the bins.
- How is model accuracy calculated? This is usually far less objective than model constructors are willing to admit to. How does your error vary along different dimensions? Which dimensional error do you look at more closely, and why?
In my experience, the best way to reason about models is to work with a lot of different models, and to be honest about their flaws. That learning generally happens more efficiently, and more broadly, in the real world than in the classroom. With the recent rise in popularity of applied models, we have lots of inexperience modelers out there, so these growing pains are to be expected.
So the problem would be: how do you get the "modelers" (i.e. the machines) to be "honest about the flaws" of the models that they generated?