I have argued in the past that data and code need to be submitted with academic papers: http://blog.jgc.org/2013/04/the-importance-of-open-code.html This is just the latest example of why, IMHO, code needs to be open.
My co-authors and I chose Nature as the target for the paper because of its reputation and because I believe in going to 11 (http://en.wikipedia.org/wiki/Up_to_eleven).
In addition, researchers are likely to discover mistakes while cleaning up the excel sheet, data sources and code before publishing it. Just like we find mistakes in our work while refactoring and cleaning up code before we push it to github.
So even when nobody ever looks at the data and code we can expect the quality of the research to improve significantly: just because the code is there to be looked at.
I think the case in favor of making data & code public for academic research is pretty overwhelming.
The more I learn about this particular incident, the more I feel the scandal is not (a) that famous harvard professors made a mistake, or (b) that microsoft excel is more error prone than alternatives (which seems like nonsense to me because the more complex the replacement the more error prone it will be)...
To me the scandal is that you can be a tenured professor in economics and produce work that amounts to simple averages of widely available data and call it a research paper, and that people take you seriously, and presidential candidates use you as a reference, and your department doesn't bat an eyelash.
The fact that they screwed up seems incidental - mistakes happen.
It seems like the kind of back-of-the envelope work that any old blogger would be capable of doing, we just don't have a way to take good ideas, no matter where they come from, seriously. no matter how much we profess to try - at heart, the consensus is still status-driven, and pedigrees matter.
Not necessarily everyone. Collecting good data is hard, long, and tedious, and the 'glory' part is the analysis. People get accolades for making analyses, not for good groundwork.
If data and code are open this is fantastic for science and progress because errors do not replicate and permute but it can be terrible for individual scientists and graduate students. As always, it comes down to money. If we are forced to correct or even worse retract a scientific publication, the community (both local and worldwide) bares down on you like a knife. The stakes are so high that not only is falsification lethal, in many cases honest mistakes can be lethal as well. Conversely one cannot take eternity flipping every stone to bulletproof every possible single problem and it can be very hard to identify which stones to check!
This is not a defense of closed work and closed data, but it's a realization that opening data and code is not a simple and straightforward process. There are severe and deeply embedded cultural problems that make post-publication sharing difficult.
An example from economics: Meese and Rogoff [1] has (by google scholar) about 3000 citations, which is massive for econ and is one of the key references in Intl Macro. They showed that all of the models in use at the time of publication sucked.
[1] Empirical exchange rate models of the seventies: do they fit out of sample? links to the paper here: http://scholar.google.com/scholar?cluster=170917352624567913...
So did I... however, showing code and data should be mandatory for acceptance of papers. Errors are all too easy to miss and since papers draw conclusions from the data a paper without data should actually be simply refused for being incomplete.
Why publications would accept a paper for review and eventual publication without giving reviewers the ability to actually check the results is a mystery to me. The current system basically amounts to 'you're going to have to believe me'. I singled out CS papers because there the code is pretty much the essence of the paper but of course the same goes for every other field.
If I brute-force detection of the first 10000 prime numbers (for some non-CS context) by testing their divisibility from n to 1 it's horribly inefficient and programmers may laugh, but it's accurate, simple, and easily reproducible. Given that a definition of a prime number as divisible only by itself and 1, it may be better to present a brute-force algorithm than to get into a sideline discussion about validating some new-fangled method like the Sieve of Erasthostenes, that upstart.
Science aims at proof rather than efficiency. What matters is the quality of the result, and its reproducibility. If you can get to the same result in a much more timely fashion that's awesome, but that's an engineering achievement rather than a scientific one. I don't think that anyone wants to put out code labeled as CRAP in an attempt to mollify engineers.
And that's to be expected, because of how most scientific code gets written, and what the incentives are. And releasing this code would still be strictly better than not releasing it. We just need to keep the expectations of code quality low, which the CRAPL license does.
Your comment reads to me like you want non-programmers to ritually humiliate themselves by labeling their stuff as CRAP before you will deign to parse it for correctness. Patronizing people is not a good way to get them on-side.
We just need to keep the expectations of code quality low, which the CRAPL license does.
The purpose of the CRAPL is to make code accessible, not to lower expectations. It's not about what you think of the code, it's about whether the code yields correct results.
Not to dredge up a functional vs imperative battle, but I feel FP has a lower impedance mismatch with mathematical concepts generally.
You might have better luck in a quasi-research public venue. For example, I work at a public agency that uses your tax dollars to forecast travel demand, and then use the results of that statistical modeling (plus a thick shmear of political wrangling) to decide how to spend more of your tax dollars.
Much of this model code is developed as part of an honest to goodness research process (yay!) by contract software developers that public agencies can afford (not so yay). In other words, things like revision control and unit tests are mostly dismissed as extravagances and unwarranted delays. Validation that is performed doesn't exactly inspire confidence.
I'm betting there are a million and one of these kind of projects. Some of us are trying to figure out internal and external issues so we can post this sort of thing on places like GitHub. Others are already there.
If you pick a topic you're interested in, and ask the right people, it'll probably be worth some authorship cred.
Does that mean to 'notify the Author about Your work', or 'notify the (Author of Your work)' (i.e. yourself)?
If this license is meant to be serious then I should probably notify its author about these concerns.
[1] http://www.aeaweb.org/aer/data.php [2] http://www.aeaweb.org/aer/pandp_faq.php
That was clearly not the case with the flawed paper this OP is concerning itself with. So, not a valid excuse in this case.
At any rate, the team should learn to write papers as they build up the data set. That way they always have a little more data than anyone else working with it. Because without release of the data on which a given paper is based, there's no way to know if the paper is actually valid.