The advantages still outweigh the disadvantages, but it's important to remember that there are always trade-offs.
The advantages still outweigh the disadvantages, but it's important to remember that there are always trade-offs.
and that brings us back to why many scientists don't publish at all now...
As long as not publishing code is standard, there's a incentive to do it since it might open you to career-withering criticism...
I have tried to explain (With limited success) to my colleagues in bioinformatics that not unit-testing your code is like using an instrument that hasn't been calibrated, and blindly trusting the results to be right.
This is the standard procedure for when you've developed a complex algorithm to do something novel, and actually working through it by hand would take hours.
I should know, this is how I used to do it before I knew better, and as a result one of my published papers has (very slight) numerical errors in.
A lot of time might be wasted going down that path too.
The quality issues aren't all that different than a web framework, either. Release one or a small number of code bases, and have everybody pound on and improve them, and you might get somewhere. Have everybody write their own code bases from scratch every time and you'll get yourself the scientific equivalent of http://osvdb.org/search?search[vuln_title]=php&search[te... .
To be honest at this point when I see a news article that says anything about a "computer model" I almost immediately tune out. The exception is that I read for some sign that the model has been verified against the real world; for instance, protein folding models don't bother me for that reason. But this is the exception, not the rule. When it became acceptable "science" to build immense computer models with no particular need to check them against reality before running off and announcing immense world-shattering results I'm not exactly sure, but it was a great loss to humanity.
In the real world, most scientists will keep modifying their program until it gets an expected result (either because the program is correct, or because multiple mistakes cancel each-other out, or the standard theory is wrong but they feel obliged to keep debugging until they can prove it to be correct). Sure, there's times when a computational model proves something interesting (unexpected), but then the researchers may have to open source it just to prove they didn't screw it up.
A lot of code is going to be like that: applying known functions that can be identified by name. If a paper says it used an FFT on some data, there are plenty of well-tested FFT implementations that can be used by another researcher to try to replicate the original result. The original researcher's FFT code, if any, isn't really necessary as long as the function, the inputs, and the outputs are well-documented.