Tried the source code on a toy model: adam took 2 epochs to train a 10k parameters model, this didn't achieve anything useful in 20.
Tweaked a bit the hyper parameters and such, but nothing. Probably a bogus implementation?
Tweaked a bit the hyper parameters and such, but nothing. Probably a bogus implementation?
My summary is that the memory savings might be great if it works, but it does not work everywhere.
Adam, on the other hand, generally gets you pretty good results without futzing too much with hyper params.
They do say it consistently matches or outperforms despite simplicity, and I think that statement means at the lower budget for their approach, but a fair comparison fk seems if it is at least promising would be take advantage of the lower memory read to add more params in their version in the comparison.
Also the paper says slow initial convergence, under limitations:
> More- over, our methods ensure a steady and stable update during training, allowing the model to converge better in a given task with sufficient training steps. Thus, we might observe that the convergence speed is relatively lower than Adam’s in the early stage of training; as our primary focus is to investigate the effectiveness of the SaI approach, we left the acceleration of convergence speed in future work.
Maybe it's just way too small, you wouldn't use Karatsuba multiplication to do 3*5.
I'm not using a transformer, just a plain Feedforward, Relu and dropout for a simple classifier.
I don't know, I can be wrong. I hope and some toy experiment shows that even in low case parameters it works fine as well as adam.