> You're double-dipping into the data. You look at the performance, then you tune some part of the network by hand, see if it helps, and then keep doing that. It's testing on the training data.

I purposely tried to avoid adding any niche network modifications that would help it overfit to in-1k. All three of the modifications are applicable to other networks and datasets.

> No one cares about ImageNet-1k. No one needs to classify ImageNet-1k in real life.

I completely agree with you; I just don't have the compute to train this on a massive dataset. With that being said, I'm not advocating for taking an in-1k model and putting it into production. I'm merely saying we can get ResNet to reach the same level of performance as ViTs. And that there's evidence that convnets reach that same level of performance at scale.

> What you want is a ViT that's seen massive amounts of data so that your embeddings don't become degenerate because they're far away from ImageNet-1k!

https://arxiv.org/pdf/2310.16764.pdf