The learning rates they demonstrate are crazy - though the standard when talking about CIFAR-10 is 94% accuracy iirc. Showing ~60% accuracy is weird.
Has DAWNBench been done with manifold Muon (with a more appropriate architecture)?
Has DAWNBench been done with manifold Muon (with a more appropriate architecture)?
Here's the top model on DAWNBench - https://github.com/apple/ml-cifar-10-faster/blob/main/fast_c...
Trains for 15 epochs and it, like all the others is a 9 layer resnet.
In fact beating SOTA is often the least interesting part of an interesting paper and the SOTA-blind reviewers often use it as a gatekeeping device.
I probably should have made the 9-layer ResNet part more, front-and-center / central to my point.