Comment by j4mie | Hacker News Reader

Comment by j4mie | Hacker News Reader

Actually, I have heard one good argument against open-sourcing scientific code [1]. It's not bullet-proof, and it won't apply in all situations, but I think there's a nugget of truth in there.

If I write a paper describing an algorithm (or process, or simulation, etc) and open source my code, someone attempting to reproduce and confirm my work is likely to take my code, run it, and (obviously) find that it agrees with my published results. No confirmation has actually taken place - errors in the code will only confirm errors in the results. Further work may then be based on this code, which will result in compound errors.

If, however, I carefully describe my algorithm and its purpose in my paper, but don't open source the code, anyone who wishes to reproduce my results will have to re-implement my code, based on my description. This is vastly more likely to highlight any bugs in my implementation and will therefore be more effective in confirming or disconfirming my findings.

I'm not sure yet what I think about this argument. It seems to only apply in certain domains and within a limited scope (what if the bug exists in my operating system? Or my math library?) but in relatively simple simulation models, it may have some validity.

What do you think?

[1] From Adrian Thompson, if you're interested: http://www.informatics.sussex.ac.uk/users/adrianth/ade.html

Your argument doesn't seem to apply to most code in natural sciences. The difference being that their theoretical models are, in most cases, only approximated by code. Yet, in papers, they claim to prove or test their theoretical model by experiments on the approximation.

Note that this is nothing terribly new. Sometimes experiments testing a hypothesis give false positives because something went wrong in the experiment. In that sense there is no real difference between a faulty thermometer and a bug in your code.

Having written this, I think your argument in fact does apply after all. In natural sciences one would argue that if your buggy code confirms an invalid hypothesis, someone redoing your argument with the same code would not uncover the problem. Publishing your code invites people to use your faulty thermomenter.

Of course I'm assuming that the published paper contains all necessary details to reproduce the experiment. In case of e.g. climate models or modelling the formation of galaxies that might be a problem because the code _is_ the experiment. Describing what code does is very hard, and it would be easier to just publish it in stead.

" Describing what code does is very hard, and it would be easier to just publish it in stead.'

Depends on what level you're describing it. "It does an FFT" or "it sorts" are pretty clear. It gets hairy if you describe the specific details of the implementations. But the implementation is likely irrelevant, because other scientists can choose any implementation. Even with complex models you ought to be able to piece much of them together out of descriptions at that higher level.

We, as tax payers, are being asked to bet over a trillion dollars on this research. It not only affects the first world economies, but it will be a serious source of problems to third world countries ("you know how we got are standard of living so high? well, you can't do that too"). The refusal to hand over code just leads to more suspicion. This is too political already and secrecy doesn't help. It is bad enough that the basic measurement methods are in dispute.

Reply on news.ycombinator.com