preetum

We have ImageNet experiments in Section 4 of the full paper: https://arxiv.org/abs/2010.08127

This is the exact comparison we make in the paper. We have subsampled ImageNet experiments as well; see Figure 3 in the full paper: https://arxiv.org/abs/2010.08127

Note that we also have ImageNet experiments, with entirely real data (non-synthetic). See Section 4, and in particular Figure 3 of the full paper: https://arxiv.org/abs/2010.08127

To clarify some other comments on this post: In all settings, we compare "Real World" and "Ideal World" for the same underlying distribution. Eg, we never compare CIFAR-10 and CIFAR-5m, we only compare "Real World CIFAR-5m" vs "Ideal World CIFAR-5m".

Oh yes, absolutely. We only make claims for "natural distributions and models" (^).

It's almost certainly possible to break this by pathological choice of a data distribution or model/initialization/optimization scheme. But, I don't think this is interesting -- what I think is interesting is that this seems to hold true in real life, in "natural settings".

---

(^) Whatever "natural distributions" means... (which I think is a good research direction in itself).

Hi, we define the interpolation threshold in Section 2 of the full paper (https://arxiv.org/abs/1912.02292) as the point when the "Effective Model Complexity" = # of train samples.

Where the "Effective Model Complexity (EMC)" of a model + training procedure (w.r.t an input distribution) is the maximum number of samples from the distribution that the model+training can fit to ~0 train error.

Our experiments are consistent with the hypothesis that the double-descent peak occurs when EMC = n; that is, when the model+training is just barely able to fit the train set.