Hacker News Reader

Comment by androidgirl | Hacker News Reader

Why isn't more research and research materials open source in the AI world? I don't really understand.

If one doesn't have your training dataset or your code, how could they possibly replicate your results?

If you have a good paper with important result, providing code and data is not necessary.

Providing the code to replicate is good form. It shows good faith and confidence. Exact replication (exactly replicating the study) is just the starting point to check that the code works and no obvious mistakes were made.

replication / reproducibility / hyperparameter sensitivity

If the research yields something really important and the method is well documented, usually it can be easily checked without having the data and the code. Things like dropout, batch normalization, residual learning, .. work over multiple different datasets and hyperparameters. You can reproduce the results without faithfully replicating the experiment.

If the claimed result vanishes unless you have the exact data, code or the hyperparameters, the research can't be said to be meaningfully reproducible in the scientific sense. Hyperparameter sensitivity is ML equivalent to P-Hacking.

how many papers important results are simply bugs?

Numerical code is already bug prone due to subtle and hard to test errors, research code that's not code reviewed or necessarily even tested for correctness can easily generate important results erroneously.

It's also counter-productive not to publish the underlying source code for these papers, as it adds a barrier to other researchers applying the algorithm in new situations. I'd be interested in seeing if those 6% of papers which include the code get more citations than the population of papers which do not include code.

> important results are simply bugs

Probably none. If the paper is important and collects citations, the algorithm is in use. Computer science != working code.

Code is required when you produce something where the scientific importance is less clear. There is need to provide more evidence. Many papers are just "Hey I made some some tweaks and it works in this particular case." Those papers should have working code.

Most papers leverage a results table which compares the newly proposed approach with existing approaches. This section is baselines the results of the new approach with prior work and helps determine whether a new result is actually important or yet another way to achieve the same results as previous work. e.g. the tables on page 7 of this paper https://www.semanticscholar.org/paper/Automatic-Acquisition-...

These tables are generated using real implementations that may or may not be correct, and should be subject to review when the paper is published.

As someone who's often implementing models from ML papers, the issue is that english-language descriptions of methods is often found to be sorely lacking. Even good authors simply forget to mention small critical details of model design or training that wouldn't remain ambiguous if they simply released their code. Part of the effort of reproducing work is certainly figuring out which aspects of the model design are critical or incidental - but this is all greatly aided by not having to guess at what was actually done from a heap of english and LaTeX generally written feverishly over a few days before a deadline!

> Part of the effort of reproducing work is certainly figuring out which aspects of the model design are critical or incidental

Isn't this the work of writing up? A paper is a claim that you have discovered something, and implicitly that other details are standard / unimportant. If it turns out that some hidden assumption was in fact doing all the work, then the claim you made was false.

I'm all in favor of sharing working code. But working code which magically does something... amounts to an anomaly awaiting an explanation. Or an advertisement.

Alas, training times are quite long... At any given time, there's probably a couple best-in-class architectures out there for the problem you're interested in, and one or two dozen interesting bells and whistles one can add as decoration, each with a paper that makes pretty reasonable arguments and has some stats demonstrating modest gains.

The right thing to do in this circumstance is an ablation study - throw together your best-possible model and then test different subsets of features sitting between your model and the 'basic' prior work. For large datasets, though, each of these models might take a very long time to train (especially if you don't work at a place with a stupid number of GPUs available).

So, lacking resources, you get your new best-ever accuracy number with your 'everything' model, and do an extensive write-up about how awesome the new bell and/or whistle that you added to the pile is... (The problem is compounded by a need to publish quick, lest someone else describe your bell/whistle first.)

Another big problem is that adding a bell/whistle to the base model often means adding more parameters to the model. There's decent evidence coming out of the AutoML world that number of parameters matters a hell of a lot more than how you arrange them. (It's real real easy to convince yourself that your clever new idea is more important than the shitpile of new parameters you've added to the model, after all.) So a really solid ablative study probably needs to scale the number of parameters in a reasonable way as you add/remove features... And there may not be obvious ways to do that smoothly.

And this is closely related to the replication study in psych: it's real real easy to do kinda sloppy work with big words attached, and convince yourself and all your peers that you're a genius.

I think a big database for reporting and searching for results with various architecture+dataset combinations would be much more useful than pushing more papers to the arxiv in many cases. (though, really, whynotboth.gif) Let me do some searches to see if a particular bell/whistle actually adds value across the god-knows-how-many-times someone's used it to train up imagenet from scratch...

Then it sounds like people are doing something quite far from science. That's not a criticism, there is lots of knowledge which isn't scientific... for example if you want to know how to run a restaurant, you need to talk to lots of people who do, and try to learn what they know about the art. Maybe you even go to trade shows and listen to talks. But if you follow what they said and you fail, it's not really a replication crisis.

It's a mix; some groups are more disciplined than others (and have better resources for ablation studies).

To be sure, some sciences are more rigorous than others. Various natural sciences are reliant on transient observations, which might or might not be 'reproducible' in any sense... And still progress is made, though the results might contain more prejudices than one would encounter in other areas.

It's also worth noting that some areas within machine learning are extremely difficult to test rigorously (e.g., generative processes), but are still totally worth pursuing. So be careful with cries of 'it's not even science, man!'

Right I genuinely don't mean that as an insult, perhaps my cooking example was flip. There is lots of genuine progress made by other means -- anyone think that Toyota's process inventions in were done by double-blind trials on a large sample of factories?

And of course some guys a few doors down are proving theorems. And lots of other things.

It is slightly strange though that we try to vet all of these with the same peer review mechanism. This is one major source of the differing opinions in this thread about how this ought to work, I think.

Reproducible builds make it much easier for others to check for things like hyperparameter sensitivity. Right?

Yes.

So the argument is that running buggy code makes your replication tainted. So ideally experiments replicate everything, meaning part of replication is implementing the code & collecting a dataset. The exact code & dataset aren't suppose to be required for the results: you should be able to replicate the results by substituting your own code & data. When you have a rat maze experiment, it isn't expected that they include a 3d printer blueprint of the maze & genomes of the rats involved

Or, more succinctly, code & data are left as an exercise for the reader

It is common to publish the layouts and dimensions of mazes in such studies. The mazes[1] themselves often reflect the nature of the study (e.g. a simple fork to test desicion-making or a complex maze with dead ends to test navigation and/or memory.)

1. http://www.ratbehavior.org/RatsAndMazes.htm

But if more journals archive artifices (code + data) and there was a bigger push to keep these around, when replication fails, they can at least go back to the exact code and say, "huh .. they implemented this differently. I think this is the problem with their algo" or "oh, we forgot to account for this." .. ideally after you've written your own, looking only at the methodology and without looking at the code for the original.

Having the journals archive artifices seems like a good idea, but the librarians hate it. The last thing they want to do is to give the for-profit journals another thing to monetize against the academy. What makes more sense is to have the universities and the library systems archive that stuff.

why not use a public artifact repository? e.g. github, bitbucket, or a new academic collaboration.

I see. That does make sense, like making a MIT version of GPL reference code.

Having more researchers avaliable to audit code seems like it would help prevent flaws from slipping through, too, to prevent false conclusions.

Thank you for explaining a bit more.

Amusingly, this is often taking to the other extreme. Just being able to run the exact same code on the exact same data doesn't really tell me much about if it replicates anywhere else.

That is, I agree with you that the code and the data should, ideally, be available. I lose confidence when people just rerun the same code on the same data. The slides a while back about why someone didn't like notebooks resonates well with me. Something like "Shift enter through the lesson. Is this learning?"

Are the 30% of papers including training data actually hosting the data set somewhere for download, or are they referencing a public data set elsewhere? Both approaches are acceptable in my book.

If a study uses a private data set, or one that the researcher controls and only gives out to approved partners, that study should be discounted.

I understand corporate labs cannot give away their data in many cases, but corporate research carries less authority than academic research anyways. Academic research should always make the data set publicly available.

I think if you're going to do a replication study, you should collect a different data set about the same subject, and use the same method, then see if you get the same results.

Yes, simply compiling the provided code and running it against provided data set does not really count as replication.

Writing your own code and creating your own dataset is the simplest way to rule out the situations where there is something fishy in either the original code or data. And the paper itself should contain enough details to make it possible to recreate the experiment this way.

I think compiling the provided code and running it against the provided data set does do something---you know the code reports on the data. If you get a different result with the provided code and data, then there's something different with your environment vs. the original researcher, perhaps the rounding mode, or some assumption [1]. Once that's straightened out, then you can use new data and see if you can replicate the result with the provided code and new data.

Think of the provided data as a sanity test.

[1] I recently fixed a bug wherein I was inadvertently relying upon Linux-specific behavior that failed when tested under Solaris.

Research is often funded by firms that want the study to serve as a “scientific” basis for the efficacy of a product or service. Experiments that cannot be replicated are meaningless. That standard should not be compromised.

There is some famous citation from a physicist about people that used to replicate each other experiments before, but now they share their fortran models, so they can agree on all the bugs.

Reply on news.ycombinator.com