Hacker News Reader

Comment by KKKKkkkk1 | Hacker News Reader

For a computer scientist, reproducibility means more work that they aren't paid to do. If I ask the Mozilla team to implement new feature X, the response will be either (a) point me to a donation link, or (b) We're open source, so why don't you implement the feature yourself? The computer scientist's response is the same.

View on HN

Reproducibility for the computer scientist means including any code written and data collected or relied on in the scientific publication itself. In practice, getting there from here isn't literally zero work, since some actual human action is needed to bundle the code and data, but that effort ought to be negligible overall, especially if we make it a standard part of the scientific process.

Trust me, it's currently far from zero work to submit code with a research paper. I was recently the corresponding author on a software paper sent to a journal that at least verifies the code compiles and runs and produces expected output. Since the poor person testing the softwares submitted is permanently in the ninth circle of dependency hell, across all platforms and libraries imaginable, it took about fifteen emails back and forth plus an OS reinstall before everything checked out. And they said that wasn't anything extraordinary.

How about a platform centered around Linux containers (or maybe one of several OS containers or VM images), as the repository image?

I'm not saying the work is zero now, but maybe we can get there. If a researcher is developing on a platform where their repository is expressed as a container-like image, then they should be able to publish it for anyone to run exactly as-is. The container repo includes the data, the operating system, and any languages and libraries, with an init system that optionally builds the results.

Yes, I think we need to go in this direction. The problem is that the container system is yet another tool for researchers to learn. The first step is to get everyone using VCS and nightly testing. Many are still at the point of clumsily written, old Fortran code that gets emailed around and exists in N different variants. (Not that there is anything wrong with Fortran.) Many are at the point where if you email them a link to a git repo to clone, they're clueless about what to do.

It would help if Git didn't have such an awful learning curve (and I say this as a git user that already went through it).

I know researchers that used Subversion when it was on the rise, but they just abandoned version control altogether when Git became the generally preferred option.

Couldn't we all just assume that the reproduction procedure starts from a fresh Ubuntu VM?

There's a difference though between "just published" and "reviewed, verified and published". We'd gain a lot if anyone simply included the code they were using with the papers. It doesn't matter if it runs and how generic it is. If it's important, it can be fixed by the next user. If it's not, it didn't matter in the first place if it's verified.

Of course verification on submission is also a great idea, but we can make it the next step.

It's more like Mozilla writing a new version of Firefox that only runs on their own machines.

This is exactly the reason. Academia is a business just like any other. Academics do what they are incentivized to do. I guarantee that if NSF, DOE, NIH and the other US government funding agencies made publishing your code a requirement for funding, and actually checked that you did, you would see everyone start doing it immediately. Right now NSF requires a "data management plan" as part of the proposal, which includes information about sharing, reusing and archiving data and code, but nobody ever checks that you followed the plan after the money is awarded.

I don't think Mozilla Corp takes donations for software development. The Mozilla Foundation does take donations, but it doesn't develop software. Rather, the foundation works on outreach (those cute videos you sometimes see on the Firefox start page).

Mozilla is a bit of a weird entity, half non-profit, half very much corporate, yet all code is free.

In theory, people shouldn't be going into research just to get paid for it.

In fact, if they are they're using a pretty ineffective path to get paid.

Good point. Research software is always far from production quality and researchers don't get into research to write production software. To publish software, it has to build and run on all reasonable platforms, and users should be able to read, understand and modify the code. All this effort distracts the researcher from doing research. (Unless we redefine research as working on software.)

Nobody is saying your research has to have the quality of industry open source projects. You're just coming up with excuses to continue to hide your code.

Merely publishing it at all would be helpful. That way other people can pick it up and modify it to work for them. That's still better than them flying blind and having to completely rewrite your code from scratch.

It's really infuriating to see researchers creating this false dichotomy between publishing production-quality software and publishing any code at all.

Yes, exactly. All I'm asking for is a literal code and data dump. Take a snapshot of the operating environment and publish it. Ideally as a virtual machine or container image, so people don't have to fiddle with dependencies and versions; but if it's just a file system dump of the code that's better than nothing.

The folks who are asking for this are NOT asking for: (1) production quality code (2) portable code (3) an open source project based on the work (one way code dump is fine) (4) well commented high quality code (5) any support using the code beyond which would normally be afforded fellow researchers in reproducing one's results.

We (me and anyone else who is asking) understand that there may be some circumstances in which not all data can be published due to privacy, or in which code depends on proprietary dependencies that can't be shared. That's fine. Document it if you can as a disclaimer, and people can work on getting access to those private resources if they need to.

Anyone funding research should expect this; and publicly funded research should be required to disclose all technical work product that's materially involved in published results.

That's all great. And when I publish my code and it doesn't work on your machine as it does on mine, what do we do next? When you're saying "should", you should say who's going to pay for this and at whose expense (in money but also in time).

If the virtual machine image you prepared works on your machine and not on mine, I'll troubleshoot it and submit a bug to the VM project. It's pretty unlikely, though - that's why virtual machines and more recently operating system containers are so useful. They are portable. You can even emulate an x86 virtual machine in your browser in JavaScript and boot Linux within it: http://jslinux.org/ - cool, huh? With mature virtualization technology it's possible to take a snapshot of an operating environment + data + program as a portable image, then someone else can boot it into a virtual machine and run it. VMs are mature and run a large portion of the software powering the Internet. Containers are fairly portable as well. See more of my other comments on this topic if you're interested in my ideas about technology specifics and how it would be funded (I'm not saying I have a full plan, just avoiding repeating myself).

Reply on news.ycombinator.com