Hacker News Reader

Comment by dontreact | Hacker News Reader

Seriously...

What's surprising is that Google has an enormous dataset called JFT where they could have tested this without this confound. Just shrink the images and you can make something cifar-like and something cifar-5m-like

View on HN

or even just an ablation on Imagenet or some other “large” dataset. Did this paper get accepted because it has five pages of citations and invokes Vapnik in the second sentence?

We have ImageNet experiments in Section 4 of the full paper: https://arxiv.org/abs/2010.08127

You're not ablating anything there. What happens when Train Infinity (Train 150K) doubles in size (to Train Infinity_2 -> 300K)? What happens when you add an unseen class? These are real-world conditions that hamper existing theoretical estimation of the generalization gap-- the "ideal world" always gets larger. In Bengio's group paper (Predicting the Generalization Gap https://arxiv.org/pdf/1810.00113.pdf ) they actually do these sorts of ablations.

Also, you use K (thousands) and $K$ (latex K) interchangeably; it's really hard to decipher is K is a variable or what you mean.

Why would you criticize them so hard for not doing imagenet experiments when they did do them?

This article was promoted by Google's PR blog and the conference is ultra selective (probably excessively selective). Hyped results deserve extra criticism. And no, they didn't do any useful ablation experiments.

Reply on news.ycombinator.com