This is a total misunderstanding of current machine learning, of how you should run experiments to avoid training on the test data, and of what performance means.

First, tuning the hyperparameters of a CNN to gain <2% improvement on a dataset when you're constantly looking at the answers is meaningless! You're double-dipping into the data. You look at the performance, then you tune some part of the network by hand, see if it helps, and then keep doing that. It's testing on the training data. This is totally statistically bogus! Never mind that the variance between runs wipes out a lot of these differences. And that this performance difference doesn't matter to any problems in the real world. The results at the end show none of this matters. This is really just the worst kind of performance hacking ML that contributes nothing at all.

Second, the point of modern ML is not to perform well on ImageNet-1k! No one cares about ImageNet-1k. No one needs to classify ImageNet-1k in real life. The goal is to generalize to new data that the customer has. For that, a ViT trained at scale is far better than a CNN trained on a small dataset.

> But small models (e.g. most embedding models) are arguably more important because of their portability and adaptability, and these models benefit greatly from the exact type experiment of outlined in this post: strong augmentation with limited data trained across many epochs. This is exactly the type of data that Imagenet-1k represents.

This is totally mathematically bogus and ignorant of like the past 10+ years of machine learning. I cannot believe that someone that has a job working in ML would write something like this. Much less someone who works at a vector database company!

ImageNet-1k represents basically garbage for embedding models. What you want is a ViT that's seen massive amounts of data so that your embeddings don't become degenerate because they're far away from ImageNet-1k! Almost everything is far away from ImageNet. It's an extremely narrow slice of the possible images.

It also confuses the size of the training data with the size of the model. Again, a confusion that would make much more sense 10 years ago. Just because a ViT has seen a lot of data doesn't mean it needs to be big; it could have started out big and been distilled down. From the NLP world we've learned that there are absolutely tiny embedding models, just a few MB in size, that work amazingly well; they can even beat multi-GB models.

As an ML researcher, I can't believe anyone would write this article today. I'd fail a student if they wrote this.