Hacker News Reader

Comment by p1esk | Hacker News Reader

Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper.

Are you sure you have thought this through?

View on HN

The experiments themselves should be conducted on fully logged public infrastructure so that it impossible to miss anything out. This final log would include the inputs: shell commands/git repositories/Dockerfiles, training and testing data; important intermediate data: Docker images and trained models; and outputs: tables of numbers or what have you. This way, the whole thing is vouched for automatically, and the editors don't need to do anything.

It would be a big change, but this is actually possible, and it would make things easier for everyone down the line.

Who is going to pay for that?

I have thought it through. There are multiple major papers that are published each year that purport to show an advancement, but fail to demonstrate that advancement under different random parameters.

This was a major topic at NIPS 2017, yet doesn't seem to be important enough to address.

8 gpus and a week is cheap even, there are popular papers that need upwards of 10k GPU hours.

Well, it usually takes months to get a paper reviewed anyway, so it's not like there isn't time for that...

It is not like, ehem, Elsevier does not have the money to pay for this...

AI papers do not get published by Elsevier

Reply on news.ycombinator.com