Actually, encountering this I think I suddenly understand why people get upset about Kickstarter projects not delivering. It is a misunderstanding of a probabilistic situation.
Actually, encountering this I think I suddenly understand why people get upset about Kickstarter projects not delivering. It is a misunderstanding of a probabilistic situation.
There is a fundamental issue here that is not being discussed is that is the science funding system incentivises people rushing results out and not check if they are real. If Julia had been more careful she would have found the results were false, not published, and not received a grant. So much of the problems with a lack of reproducibility in science is the direct result of people rushing to get out “novel” data and not checking if it is real because if they do they don’t get funded.
Yes, someone whose conclusion was not due to error in programming may have gone without a grant. That's perfectly all right. We optimize policy in the aggregate.
You can make an argument that says that we're not correctly placed on the manifold but no single situation will be convincing for that and you'll need to make some case that a policy change to move to some other point in this space will yield a positive total improvement.
What you are asking for is the equivalent of saying "firing programmers when a bug is found will result in better software"
In science it will result in much less interesting science being done, because this will desincentivize risk.
The big problem in science funding is that it already incentivizes low risk projects (you should have preliminary results that proof it works...) so let's not make it worse
From a software perspective it is like paying developers by the number of lines of code delivered and telling them to not worry about bugs. Not an approach that is likely to deliver quality software.
What we need in science is to find a way that people have the time and incentives to publish results that are as accurate as possible.
Somewhat off topic I do wonder if we would get better software if we fired programmers if bugs were found. I suspect we would not get much code written, but the code that was written would be very high quality.