For anyone else who isn't familiar with what CIFAR-10[0] is about.
I imagine this means that the entire training set was ingested in 12.4 sec.
I imagine this means that the entire training set was ingested in 12.4 sec.
Previously a number of years ago (maybe 4-5), the world record was ~10 minutes, so quite a lot of increase! I've picked up the torch as nearly everyone has moved on from it, but finishing a near-optimal version of this benchmark will transfer to transformers and whatever training techniques come after them in the future.
I hope to do similarly for language models in the future, but writing it from scratch (I've only started) is a haul. But this is also an entirely hackable codebase, so you can change anything from the dataloader to the final SGD step call within a few minutes (without having to do several minutes - hours of debugging or rewriting).
I really appreciate you linking the CIFAR10 explanation.
The irony of pointing out missing context and providing insufficient context myself.
96% is another candidate but much harder, and I think robs the 4-6x speedup in experimentation cycle times that 94% requires.