Comment by noname123 | Hacker News Reader

Comment by noname123 | Hacker News Reader

Parent Full thread

>Failure to distribute the data and code not only greatly devalues the contributions of many scientists, it makes replication far more difficult, and opens the door for outright fabrication.

Reasons why labs don't produce code:

(1) Fear of being scooped; suppose you put in tons of man-hours into a custom population genetics association study for malaria in Africa, you don't want your competitor lab to sequence a bunch samples in SE Asia and run your code as-is and publish a paper when you have the sequencers to do the same.

(2) Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results, e.g., (https://liorpachter.wordpress.com/2014/02/11/the-network-non...)

(3) Focus on science, not tool production; most labs' focus is to produce publications, not open-source software. User base for scientific software is very niche, unlike Web MVC frameworks; making the payoff calculus of packaging and supporting external users not so great. Furthermore, most labs have custom databases making integrating external software, externalizing internal software difficult.

(1) You publish the code at the time of publication of the paper, not before. (2) This is a reason to publish the code, not a reason not to publish it. (3) You don't necessarily have to make package it neatly like production software, but some effort is necessary.

We just wouldn't accept these kind of excuses for other descriptions of methodology, even though they might apply. There's no reason that code should be an exception.

For 1, I read that as thinking you could get a second paper out of running the same code on another dataset, and not wanting to be scooped on the second paper.

I understand the thinking but this should sound as non-sensical as

> We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work!

We didn't release our first, second or third papers because we didn't want to be scooped on the fourth and if we publish then someone might build on our work!

Which happens all the time. If you are working on something big you'll sometimes hold back on publishing parts of your research until you've completed the big thing for just that reason. Then once you've completed the big thing you publish all four papers either together or in quick succession.

Just running the same code on another dataset is not research, it's production. Without minimal extra scientific efforts, such a work, while valuable, certainly does not deserve a scientific article in a peer review journal.

>Fear of competitors not being able to replicate stochastic ML results; some machine-learning and artificial neural network papers applied to Biology are stochastic. Past authors have been accused of "cherry-picking" the most "optimistic" runs that show positive results

That sounds like an ML phrasing for "fear of non-reproducibility", which is basically a fear of their research being bad.

Just to follow up on your last point, I have found that folks are often happy to send code, but code written in research labs is generally horrendous. I don't think that the "publish or die" culture and a culture of releasing all relevant code is compatible. It takes time to refactor and produce clean code - time that can be spent chasing the next publication.

Then requiring the code to be published and runnable would have a salutary effect on the whole circus.

Not really... I don't see how that would be anything more than adding one more task to a million other tasks. It doesn't change incentives.

I don't see the need for clean code, I just want the code they used. I've repeatedly tried to find the code used to generate various visualisations (which were the entire point of the paper) and then found

1. "It'll be released soon" on a years old page 2. 404'ing personal pages on university sites 3. Finally some code

But then the code didn't, and couldn't, generate the images they have. So now I'm not actually sure if the algorithm works as they say.

This was also one of the more successful results I've had in trying to find research code outside of ML. I'm sure other disciplines are also good but ML is mostly what I've had to find.

There's a part of the experiment which virtually anyone could replicate for a material cost of near 0 and check in detail if they wanted to and yet that seems to be the most poorly explained and shared part of it. You wouldn't get away with just mentioning the protocol you used in the lab without any references or explanations, would you?

For academic labs you can add 'perceived future commercial value' to that list. In my experience these groups are often run by people with little or no commercial experience and lack the skills to appreciate the difference between a prototype and a commercial solution. Coupled with the pressure from Universities to 'commercialize all the things' this can inhibit attempts to distribute the code freely.

>Fear of being scooped;

This prevents a lot of good research from coming out. We should not just rely on publish or perish. Ifa lab makes a net contribution, they should still be rewarded with grants even if the idea gets "scooped".

>replicate stochastic ML results

(1) use an off the shelf prng

(2) provide the data

(3) provide the seed

Reply on news.ycombinator.com