I really like the idea at the end of using prediction markets to figure out which studies should be challenged by attempting replication.
Attempting to replace the role of independent replication with opinion is an awful idea. And that is the goal here, to replace, not to supplement. Of course, it doesn't seem replication attempts are very common in this area to begin with. So this is actually going to be a justification for continuing that pseudoscientific practice and to avoid checking all the previous results.
Suppose there is a parameter - say the probability q of a Bernoulli random variable being true. Based on your past experiments you have a posterior p(q). Then given p(q), you can easily compute a probability distribution on S(N), where S(N) is the number of successes you'd get from a bernoulli random variable with N attempts.
In code terms, you are just computing posterior.flatMap(q => Bernoulli(N,q)). (Using the inherent monadic structure of probability.)
This actually works in general. If you want to predict the outcome of a later experiment (the replication) you just compute posterior.flatMap(parameter => generateResult(parameter)).
Then you can simulate the results of future experiments with _n_ datapoints: sample 1 effect size from that distribution, generate _n_ datapoints assuming that effect size, run the hypothesis-testing, and return the _p_-value.
The fraction of _p_<=0.05 is your best forecast of whether the future experiment will succeed in reproducing it or not.
If you successfully replicate, then it has survived the challenge. The more challenges it survives, the more confidence we can have that the result is valid.
That said, we do not try to make replication challenging. We try to present experiments in a way that makes replication as easy as possible to perform. Exactly so that people don't have to take what we said on faith.
The point of the prediction market here is not to replace independent replication with opinion. It is to ensure that the energy that gets spent on replication is more likely to be spent effectively.
Ideally, of course, targeting replication efforts more effectively will increase the value of time spent in attempting replication. This should therefore increase how much effort is spent on replication. Which is exactly the opposite of replacing independent replication with opinion!
A lone report of some observation should never be believed. It should be verified by others who retrace the steps.
>"If you successfully replicate, then it has survived the challenge. The more challenges it survives, the more confidence we can have that the result is valid."
If the observation can be independently replicated, it shows that the methods required are understood well enough to communicate and that it is stable in the face of unknown influences. This increases our confidence that we understand what is going on and that the phenomenon is worth theorizing about.
It has nothing to do with an observation being true or valid. The observation was made, it happened. It is true. It is valid. (Excepting outright fraud, which is treated equivalently to some severely unstable phenomenon)
>"The point of the prediction market here is not to replace independent replication with opinion."
They explicitly say that is the goal in the paper. I like this method of eliciting priors, not that the goal is to substitute it for actual replications.
>"It is to ensure that the energy that gets spent on replication is more likely to be spent effectively."
If a study isn't worth replicating, then it wasn't worth doing and reporting in the first place.
See http://www.the-scientist.com/?articles.view/articleNo/43875/... for horrible replication rates in psychology. See http://journals.plos.org/plosmedicine/article?id=10.1371/jou... for evidence that medicine is no better. See http://www.nature.com/nature/journal/v483/n7391/full/483531a... for evidence that the research on which we base clinical drug studies is also in bad shape.
At this point anything that focuses attention on, "We should replicate SOMETHING" is a huge improvement. Eventually we should get to, "You should expect to be replicated." But we are a long, loong, loooong ways away from that now.
I am well aware of this problem. I expect you will have no more success convincing these people that replication is necessary than you would convincing a fervent religious believer their deity does not exist.
These juvenile research practices really need to stop though. It is driving the most intelligent and competent potential contributors away from careers in these areas.
This. Null hypothesis testing: Just stop it. We've known this is a bad idea for decades