The Effect of Statistical Training on the Evaluation of Evidence (2016) [pdf]

The issue really surfaces when you dissect "statistical training". I had a conversation with a colleague getting the a PhD and about to publish a paper. She was mentioning to me that she tried "a bunch" of different specifications and the one that was significant was [I forget what the variables were, but this is in the education field].

I pointed out to her that, since she tried a bunch of specifications and chose the one which yielded the "positive" result, wouldn't that fundamentally alter the interpretation of the p-value, and possibly invalidate her claim?

She looked at me like I was a moron, uneducated, and trying to be difficult.

I'm forever indebted to my econometrics professor who helped me build a tremendous mental model about how to think about these issues. But between the really poor training (most people I encounter in the field just run STATA/SPSS commands with limited/no understanding of what they do mathematically) and the terrible incentives facing researchers (you don't see many negative results published, do you), it's no surprise that when people come in and attempt to seriously replicate even foundational findings, they often come away disappointed (thinking about the Reproducibility project - https://en.wikipedia.org/wiki/Reproducibility_Project)

I, too, don't have an answer to this. But I can only imagine that the way to prevent is to push the industry in the direction that truth-seeking, not novelty, is rewarded. I've mentioned this before here, but I know professors who established their entire careers on the backs of research which was later completely refuted -- even in cases where their actions where fradulent. And they are still teaching -- and successful.

As academics - like media and enveryone else -- are chasing our ever-dimming attention spans, I think this'll be a really tough nut to meaningfully crack

Why can't journals set a bar and flag some of these practices in peer review?

It's disillusioning to see that academia seems to face as many of these issues as any other profession or industry.

So much productivity is lost due to behavior driven by long standing incentives to benefit institutions, corporations, and established power, rather than incentives that hold people accountable for quality and the efficient acquisition of knowledge.

Is it so much different than problems that come up with politicians, police officers, medical doctors, or corporate bad behavior?

I think my naivete was that a community dedicated to truth and discovery would somehow be less susceptible to these problems, when in fact, it's just human nature reacting to an environment that can be as unhealthy as any other.

At least I can allow myself an occasional fantasy of earning the 30B I'd need to take over Elsevier and turn it into a non-profit interested only affecting positive change.

At least speaking as a academic (4th year PhD student), one of the challenges with researchers over reliance on NHST seems to be the apparent lack of a more compelling alternative. One candidate which is gaining traction is Bayes factors but there are challenges with this approach as well e.g. with suitable specification of priors. The best way forward will involve fundamentally restructuring how we educate incoming researchers because it will necessarily mean embracing uncertainty over dichotomization of results into yes/no. Andrew Gelman and John Carlin write elegantly about ways forward here: http://www.stat.columbia.edu/~gelman/research/published/jasa...

Whoever choose letter p (suggesting probability) over s (significance) or e1 (type 1 error value) did a major disservice to the world of science.

P value is a measure of error (And not a probability but maybe one kind of likelihood), not evidence, how likely it is we are seeing the result of a chance. (How the chance is defined is important and hidden in the method used to compute the p value.) And any hard cutoff is really missing the point, there is a reason this is a number not a binary value.

Effect size estimate or risk ratio are much more useful numbers anyway.

I guess this could be considered kind of OT. On the other hand, if people were introduced to it sooner, before and outside of the context of a personal investment, i.e. "proving" their own research, we might be better off also with regard to our science.

----

Delaying statistics until college -- for most public school programs -- is a mistake.

It leaves a large portion of the U.S. population with a fundamental mathematical illiteracy.

And given the degree to which statistics rule our lives and collective decisions these days, a societal illiteracy.

Just look around you.

Okay, this is really interesting. I'm pretty skeptical of their methodology for assessing "statistical training." The setup could result in studying a different correlation entirely. For example, we might instead be studying the difference in a researcher who's unwilling to send a survey back without Googling or double-checking the work vs. a researcher who is busy or willing to be wrong... among researchers who are willing to respond in the first place. I don't expect that to be a productive line of discussion, though, so I'll try to ignore it.

I can't help but think of Thomas Kuhn, who argued that the institutions of science (from researchers to reviewers to the press) tend to ignore study results that conflict with the current paradigm. So as long as our paradigm is correct, science progresses extremely quickly. When our paradigm and assumptions are wrong we spend a lot of time floundering because we ignore conflicting data that should instead lead to refinements or more questions. We want to slap a label of right/wrong on something and move on to the next question. That mentality can really hinder our ability to assess anomalies later on.

This is similar to the problem discussed in the paper. When researchers decide a paradigm or test result is true or false with no further thought to the confidence level or details, you end up with a system where uncertainty and anomaly are both ignored. It then takes a tremendous amount of momentum to overcome the assumptions, which are sometimes several steps back into "accepted science" at that point. We might even throw out some good ideas from the previous paradigm as we transition. I'm not a physicist, but it seems like they're seeing this happen right now. We've certainly seen this several times with dietary health.

A decent analogy might be a hike with unclear trails and markers. At the first crossroads we might decide that left is the correct path to your destination. Once down that path, there are dozens of other paths. If we find trails that continually lead nowhere, the common human response is to keep trying well past the point where we should have gone back to our original assumption. When we finally decide to go back and try the right-side path we completely give up on the left side, despite the fact that we haven't checked every possible sub-trail on the left side.

Wandering may be impossible to avoid, even with good judgement, but we can avoid a lot of wasted time by looking back at each of our crossroads/assumptions, assessing the probability that each is correct, then moving forward in a way that's most likely to answer a new question while testing the previous assumptions. By and large this is not how science is working. We get a few studies, accept something is true, and then wander off randomly down the trails that look most interesting.