Hacker News Reader

Comment by qwerta | Hacker News Reader

My teachers always told me that main criteria in "science" is possibility to verify and reproduce. Study which relies on statistics, but no raw data or code is presented, is not scientific.

It does not take millions to check for basic mistakes. One person with computer and free afternoon is enough.

It is like heaving open-source, but without any source code.

View on HN

While I think availability of data and code is important, note that it does not accomplish many of the functions of replication of experiments. If two people implement the described algorithm and get the same results, we can be a lot more confident than if two people run the same implementation, because the same implementation is more likely to have the same bugs.

Code has to be reviewed and _VERIFIED_. How can an article pass peer review, if nobody even checked code for basic mistakes?

Yes, precisely. Publishing code and data is necessary for peer review, not for replication. Both are necessary parts of science.

By this criteria, there is very little science going on anywhere. We don't teach philosophy of science enough anymore. In most of the fields that I'm familiar with, reproducibility is very hard and rarely bothered about by either the original researcher or others in the field.

That's true, and it's a problem.

Agreed, but I'm not sure the solution is upping the statistical standards so much as educating researchers about what the statistical standards actually mean, how to use them, and how to construct a valid, repeatable experiment.

And, importantly, providing more incentive to actually attempt replication.

Reply on news.ycombinator.com