Hacker News Reader

Comment by worik | Hacker News Reader

This is why publishing code and data, together, is so important.

Irreproducibile results in computing have no justification in this day and age.

If you run their code on their data you haven't reproduced their result. You have merely copied it. If their code works on another very similar data set to produce a very similar result, then you have something. Or if you're trying to understand the technique, you have to write your code and see if it reproduces their result on their data.

What nonsense! Copying something is reproducing it. If it cannot be copied then it cannot be reproduced. If it can be copied then it can be investigated.

I ran into exactly this problem with:

    White's Reality Check (WRC), described in White's (2000) Econometrica paper titled "A Reality Check for Data Snooping",

The algorithms that were being tested were so poorly described that less than half could be reproduced.

Once we got there for what we could we found two fundamental problems with "Whites Reality Check", one theoretical that is blindingly obvious once you see it and the other that they ignored trading costs. When we measured the levels of trading costs that would permit the algorithms to work it was clear they could not.

That we could not deduce without reproducing the results. If we could have copied the original code it would have shaved about eight weeks from my research.

BTW "Whites Reality Check" was at the time pone of the most cited paper in its, rather narrow, field.

But running their code, presumably, would get their result on their data, unless it was a fraud or weird misconfiguration. We’re not talking about fraud, we’re talking about reproducibility supporting a theory.

You suspected the result was wrong, so your interest was in debugging their code to contradict their result. That has value but also is not a reproduction of the study.

> If you run their code on their data you haven't reproduced their result. You have merely copied it.

That's like saying you can't learn from exercises in the book if the answers are in the back.

You certainly don't learn anything by running `sh run.sh`.

You learn if it's deterministic or not under the data given as input and the operating environment.

you learn run.sh actually works

Irreproducibility can result from irreplicability or many other reasons. Publishing the code & data helps rule out the trivial replicability failure: that the authors made an implementation error, so the result is not reproducible. Establishing replicability is essential, but that won't prevent all irreproducible results. Attempts to reproduce results under different conditions are still necessary for the functioning of the scientific process - but they are a waste of time if the original result is not replicable.

If you agree that publicly funded code must be free, sign the petition: https://publiccode.eu.

This is the main reason why we are building Nextjournal [0]. By using immutability of Datomic + Clojure the code (and the results) are immutable and permanent.

[0] https://nextjournal.com

Irreproducibile results should have zero place in science or academia. Hell, it has no place in politics, the media, etc. "Irreproducibile results" defeats the purpose of proving that it is not bullshit or a farce.

Reply on news.ycombinator.com