521 karma · joined May 3, 2019
This is the number one misunderstanding when it comes to Bayesian stats. Priors are hard, priors are often bullshit, priors are often the source of a “garbage in garbage out” problem, absolutely. I don’t mean to suggest Bayesian stats as something magical (magical thinking will get you in trouble). But whenever a statement like this is made, the implication is that there is some alternative where we can solve the same problem but without priors. That’s just not true: priors are an unavoidable fact of life. If you’re not explicit about your prior, it means you’re still using one but not being upfront about it. So I would agree that priors are difficult and problematic, but they are also unavoidable, and I would not say they’re “ad hoc”. I would also say it’s important to understand what they are.
> I'd even say the deterministic view is dominant currently. So yes by thinking differently one can get intuition beyond the common knowledge. But it's a nice-to-have not a necessity.
I don’t know what you mean by “deterministic”...do you mean “frequentist”? If so I would disagree completely. Frequentist and Bayesian views are equivalent except for philosophy, and frequentist stats are taught at all levels of school until grad school (at least in my experience) and I think that’s a huge mistake. What do you mean by “nice to have not a necessity”? If you’re solving a statistical question, stats are a necessity. Other fields are the nice-to-have intuition. I would agree however that sometimes you’re solving a NON-stats problem in which case have at it with whatever field makes sense.
> And one can do statistics without being a Bayesian of course.
Again, fine, I agree you can use the same math with a different philosophy, the philosophy is up to you, but if you think somehow you can do inference without priors I’m sorry but that’s wrong. In my experience “Frequentist” usually has meant Bayesian but with a flat prior (please comment if you have a counter example).
In summary: study what you want, and lots of perspectives bring more understanding, absolutely. But I stand by the importance of understanding Bayesian stats for doing ML. Even if you don’t like Bayesian stats, it’s still important to understand what is going on. Also I should be clear by “Bayesian” I mean nothing more than understanding what posteriors, priors, and likelihoods are, not a hierarchical model with MCMC or something.
All of this is opinion for the most part, and if you feel there is more to learn from alternative interpretations, fine, but the suggestion is to understand the fundamentals of what you’re doing, and you’re usually doing statistics in ML/DS (whether you know it or not). Also, understanding Bayesian stats will make your life easier and it will make it easier to understand lots of other ML concepts in a unified way rather than in an ad hoc way: “minimum length solution” or “sparse solution” is what I mean by ad hoc. Both of those things are true and important, but they’re ad hoc.
Ah I see yea, if you’re talking about hyperpriors and marginalization and MCMC I totally agree; these are really valuable in science, and this is the “full Bayesian solution” but with many caveats, one big one being that it’s very easy to be overly confident in the results of some unwieldy hierarchical model and ignore the fact that the priors (or hyperpriors) are often times fudge factors because it’s so damn hard to articulate your prior belief as a bona fide distribution. A lot of times you run into a “garbage in garbage out” problem and it might not be obvious right away since we are probably busy patting ourselves on the back and popping champagne bottles because we’ve gotten our MCMC chains to converge.
> For example Ridge regression where you put a gaussian prior on the predictor then guess at its variance as a hyperparameter for a quadratic penalty (with cross validation or something) is MAP but not actually "Bayesian". Bayesians also put a prior distribution on the hyperparameter itself making life extra difficult.
Yea right — nothing wrong with cross validation in a lot of cases. Since yea we could go with the “full Bayesian solution” and marginalize over some distribution of our hyperprior, but that very well might not give you anything more than cross validation (or maybe will give you something worse if you have a bad hyperprior).
> Even in deep learning it's simple to put a penalty term on the loss or weights for that matter, so you could call it a MAP estimate. That's implemented everywhere.
Agreed, my only argument is that it’s helpful to know where all of these terms come from in a general sense rather than treat each one as some ad hoc solution.
> For that matter you can claim any neural net classifier with a sigmoid or softmax at the final layer is doing logistic regression. With multilayer networks just adding a representation learning stage to it.
Agree and that’s exactly the sort of valuable insight that’s helpful to have (though maybe this one isn’t really an insight from Bayesian stats).
> As I said cheers to statistics, just that stopping at MAP estimates will cover most everything. If one goes looking for a Bayesian methods text or review article it will be well beyond that.
Sure, if you know MAP and what it means, then you know what a posterior means and you know that every model you fit and loss function you use comes from that, and you know if you care about more than a mode or mean of the posterior you can use lots of tools to deal with that. I’ve never really encountered MCMC in the work I do and i don’t think it would really bring any additional insight into what we’re doing, but sometimes we care about uncertainty and Laplace approximation or variational inference can do the job just fine.
> By scaling problems I basically the fact that monte carlo doesn't work in large numbers of dimensions, where the parameters of the joint distribution of everything are vast.
While I would agree MCMC and it’s variants are certainly not a good idea a lot of the time because they require a lot of time and computation, very rarely they are (you want insights from your data to use internally or something and you really want to explore implications of many different assumptions), and HMC etcetera can deal with very high dimensional settings (> millions of parameters). But generally I agree, if you’re talking about “scalable solutions” it usually means MCMC is not really what you want to be doing.
* do you want to have your NN model your uncertainty as well as the mean? How do you incorporate that into the loss function? Hint: loss = (yhat- y)^2/sigma_hat^2 is missing a term but you wouldn’t know that if you don’t come from Bayesian stats.
* the rabbit hole goes as deep as you want. Understanding Bayesian stats removes a lot of the “ad hoc” and intuitive guesswork that goes into ML when you don’t have a solid statistical foundation for what you’re doing.
NN’s are universal function approximators. They can have arbitrary model capacity, and you can sort of control that with architecture decisions, loss function/regularization choices, and early stopping, but depending on the problem they can cause more problems than they solve. Usually you don’t really know if your NN will generalize well outside of your train/test distributions, so many times it’s better to have a simpler, more predictable model that you can control the behavior of. This is all from my personal experience and is completely moot when we’re talking about e.g. NLP or vision tasks or situations where you’re drowning in data. NNs are super interesting and powerful, don’t mean to suggest otherwise but the mantra is: “what is the right solution to my problem”. Lots of great advantages to NN’s as well (you can get them to do anything with enough cajoling and they can be solutions to major headaches you would usually have in e.g. kernel methods).
I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean).
> Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling
Can you be more specific here? It sounds like you’re talking about a particular problem or class of methods. PGMs/Bayesian methods can mean basically anything from logistic regression to running HMC on some hierarchical model using 10,000 CPU hours. I just mean aspiring to learn PGMs will force you to quickly learn and gain a deeper understanding of and appreciation for Bayesian stats, and then you can build on that for years and years. But it depends on what you’re interested in doing —- there’s a difference between model building and inference; you can spend your whole life using the same loss function and just focus on making your NN architecture better, you don’t need much Bayesian stats to do that.
> i.e. "MAP" estimation which is more of a hack to ML than an actual Bayesian method
Huh? Maybe we mean different things by Bayesian — the mode of your posterior seems pretty Bayesian to me!
> Meanwhile a strong basis in "deterministic methods", as an alternative way to spend that learning effort, has its own rewards. The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. For that matter a thorough understanding of SVM delves into convex optimization, an extremely powerful framework as well.
Would agree that optimization is an important part of ML/DS, but since nowadays virtually all of the most popular optimization algorithms are available at our fingertips in e.g. pytorch, I would still think its better to start with trying to build a fundamental understanding of how to frame problems. But that’s colored by my own experience and background, people’s priorities should be different depending on what they want to do.
Bishop's Pattern Recognition and Machine Learning has a chapter thats free online: https://www.microsoft.com/en-us/research/wp-content/uploads/...
Bishop's "Pattern Recognition and Machine Learning" has a chapter on PGM's that's free online: https://www.microsoft.com/en-us/research/wp-content/uploads/...
Murphy's "Machine Learning: A Probabilistic Perspective" is another behemoth that covers this stuff, but it's really just your preference.
I say "aspire" because (1) depending on your background, it will likely be something that takes awhile to internalize and really understand, and you will probably realize many times over that you thought you understood something that you actually didn't (2) by learning PGM's, you learn a lot of Bayesian statistics as a side effect, hence why even learning a little bit about them is rewarding.
Once you learn a bit, I would use Pyro/other libraries and try to actually build PGM's for toy problems (or non-toy problems too..) because (1) it will force you to admit to yourself that you don't understand something, (2) the documentation for a lot of these libraries is also useful learning material, and (3) you will see once you learn these libraries that it is fairly easy to do something that would be astoundingly complex if you were to try and do it by hand.
You can basically build most standard ML algorithms as a PGM, so e.g. you can try to do logistic regression as a PGM and compare the results to scikit-learn.
In particular, aspire to learn probabilistic graphical models + the libraries to train them (like pyro, tensorflow probability, Edward, Stan). They have a steep learning curve, especially if you're new to the game, but the reward is great.
All of these methods have their place. SVM's have their place, but also aren't great for probability calibration and non-linear SVM's like every single kernel method can scale absolutely terribly. Neural networks have their place, sometimes as a component of a larger statistical model, sometimes as a feature selector, sometimes in and of themselves. They're also very often the wrong choice for a problem.
Don't fall into the beginner trap: sometimes people tend to mistake 'what is the hottest research topic' for 'what is the right solution to my problem given my constraints, (data limitations, time limitations, skill limitations, etc.)'. Be realistic, don't use magical thinking, and have a strong basis in statistics to weed out the beautiful non-bullshit from the bullshit that is frustratingly prevalent (everyone and their mother is an ML expert today).
EDIT: I want to also clarify: I don't mean to suggest the author is new to ML, I just mean this as general advice for anyone coming here who is new to DS/ML. The article looks great!
Our goal is to simply avoid hospitals becoming inundated. How would it be possible to flatten the curve to "slightly below hospital capacity"? To do that we would need to know exact numbers on hospitalization rates from infections, have a testing capability that is far beyond what we currently can do, and then we would need to have fine-tuned control on peoples' behaviors and also never be wrong. We have too much ignorance about too many things to do this in a way that you would deem optimal. This is a disease that takes a median of 5 days to incubate, so as soon as we get something wrong (hint: we will get it wrong), it festers for 5 entire days before we know it, and then we're stuck with the consequences. The only rational choice is to take severe action and hope it's enough. It wasn't enough in Italy, it wasn't enough in NYC.
If you're saying that social distancing/lockdown policies are an "overreaction" because we still have ICU beds and ventilators, I think that's a pretty good sign. The entire point is to do something drastic now, and gradually ease distancing measures as it becomes safe to do so without causing additional large-scale outbreaks. As soon as we have the ability to contact-trace all new infections and can successfully contain outbreaks, we can start letting up.
> Data is showing that the infection fatality rate is around 0.5%-1%, and is concentrated in older people.
right, "old" (65+?) people are going to die anyway, lets just let them die sooner. How much sooner? Years? Decades? Does this really sound like a cogent counterargument to not letting people die?
Let's ignore the morality aspect of this, which I don't think is in your favor, to put it mildly. I don't really like people suggesting that millions of people should die because "this is sad, but at some point, everyone has to die." Yikes.
But lets pretend we live in a very nauseating reality where older people don't really matter very much. You should still want society shut down to prevent this spreading out of hand because this will easily and thoroughly overwhelm all of our healthcare resources, which will mean hundreds of thousands of people you actually do care about (i.e. non-older people) will also die from lack of medical care either from COVID, pre-existing medical conditions, new medical conditions, etc.
Yea, but COVID has the potential to kill a lot of people, quickly -- are you suggesting it's a bad idea to "shut down society" to keep our hospitals functioning? I get the point that the economic cost is severe and also comes with its own share of human cost, but we're talking about saving ~1-2 million people in the US alone by doing this. Several trillion dollars is still worth it...
(1) you are overwhelmingly unlikely to actually have the disease even if your test comes back positive but you don’t have obvious symptoms
(2) the pool of people testing positive is completely dominated by false positives. The fraction of people testing positive that will actually have the disease would be very small, so you lose the epidemiological benefit of the test (now you can’t actually track the disease well).
So — test is useless for people like you and me who don’t have symptoms or a strong prior probability of having covid BEFORE the test, because even a positive result is overwhelmingly more likely to be wrong, AND now the epidemiologists trying to actually track the disease are instead overwhelmed with false positives so the true signal disappears. It’s bad all around. You could imagine using this as a screening tool, but the false positive rate has to be small enough for that purpose as well (like, less than one percent).
Let’s say there’s 100,000 people in the US with covid right now (like 10x the measured number)
FPR =0.5 TPR = 1
Disease rate = 1e5/3.3e8 = 3e-4
P(you have covid| test is positive) = 3e-4/(0.5(1-3e-4) + 3e-4) = 0.06%!!
This is less than useless. Now instead of tracking the disease you are completely overwhelmed with false positives and the result is exactly the same — everyone stay home.
That’s a ramble, but basically I sincerely hope your optimism is correct here, but I am much more pessimistic. I think we will get through this but at a huge economic cost.
- the fatality rate for covid is high. People make a valid point about how we don’t know the “actual” (unbiased) fatality rate, that may be hard to estimate, but at this point it’s more or less irrelevant —- we already know for a fact that it is high enough to overwhelm the healthcare system, as is already happening elsewhere. We’re not going to wake up one day and say “oh oops, the fatality rate is only 0.03%, oopsies sorry everyone go back to work”, it’s already past that point.
- because it will overwhelm our healthcare system, countries have no choice but to impose extreme measures to slow the spread, which will have huge economic effects depending on how long we can stand doing this. Ideally until there is a solution to the problem of inundated hospitals, whatever that looks like, and however long that takes.
- people hoarding resources or thinking this is the apocalypse are likely overreacting. People are panicking, which will slowly stop once people adjust to their new temporary reality, but whether people are “overreacting” has a different answer for every person.
- the impact on the economy is going to be severe, even with trillions of dollars in stimulus. The restaurant/entertainment industries are trillions of dollars in size, so this will hurt (it is already hurting) and it will put many people out of business.
So....it depends on exactly what you mean by overreacting. If people who are really ok are filling hospital beds, then once things get bad they will be sidelined in favor of serious cases. So for now I don’t see how that’s an issue but maybe it is?
In summary: panic is usually the wrong response to anything but this is incredibly serious...
I won't delete the comment for now, but generally I am trying to be wary of giving actual advice -- I do know stats, but I am not an epidemiologist and I definitely don't want to give out wrong information during a time like this.
FPR: 1% P(covid|+) = 2.3%
FPR: 0.1% P(covid|+) = 23%
FPR: 0.01% P(covid|+) = 75%
FPR: 0.001% P(covid|+) = 97%