Hacker News Reader

Comment by tw1010 | Hacker News Reader

This is only going to get worse. I see literally no counter force putting any kind of barrier against the incentive to e.g. overfit in order to get exaggerated results. The lucrative reward is too large not to fall for the use of shady tactics (which are sometimes not even consciously obvious; not being critical enough of your own work is a form of model loosening which acts in favour of more impressive results).

View on HN

A major factor is that machine learning is so conference deadline driven. Does the main result look good enough? Submit it! Ablation studies are the last thing you do, and often you are pressed for time due to the deadline. Since so much work is done by PhD students, they are incentivized to get their work out as fast as possible because they need to graduate and papers in top-conferences play a huge factor in getting jobs in academia and industry.

I'm aware of multiple papers by top labs where their state-of-the-art results are really the result of some very minor change (e.g., some pre-processing technique, bigger images) that was swept under the rug in their paper. All the math and the complex model had minimal impact on the actual numeric result, with the actual reason being unpublishable in a top venue.

Your entire first paragraph makes me cringe. Coming from an academic/research background, all of the things you mentioned goes against everything I was taught in terms of being rigorous with your research.

Is this more evidence we have really entered the "click baiting" era? Where the end result of the research is now secondary to getting published and into big conferences to get some notoriety?

  all of the things you mentioned goes against
  everything I was taught in terms of being
  rigorous with your research.

There are certain areas where "the system" asks people to do things right - but provides vast rewards for doing things wrong, if you can avoid detection.

If you're a pro cyclist who isn't doping, good for you! But there's no medal for finishing fourth; you'll be rewarded with medals and sponsorship cash for doping undetectably.

Academics whose rigorous papers aren't quite getting into top-tier journals are in a similar situation.

I also struggle to understand why PhD led research in AI is considered so highly. Only very rarely does is hold a candle to team lead research in the private sector.

Didnt you make the exact same post here? https://news.ycombinator.com/item?id=17497235

I disagree. There is a cost in publishing bad work, and an incentive for 'predators'. For example, MIT published a bad paper at this ICML, and I was incentivized to tell it publicly: https://medium.com/the-ai-lab/mit-paper-in-ai-for-drug-disco...

Reply on news.ycombinator.com