It's not. That's an old picture from last year.
1,103 karma · joined March 22, 2009
> Using four sampling steps, ADD-XL outperforms its teacher model SDXL-Base at a resolution of 512^2 px
Can it be any more of an obvious scam and/or ramblings of a cryptography kook.
Forward-pass only doesn't mean it's faster. It converges much slower than even SGD. It is a memory-time tradeoff, but that's it.
"As a limitation, MeZO takes many steps in order to achieve strong performance."
I don't see what you base this on. MeZO trades one back-propagation pass for another forward pass. Why would that be 12x faster? It's also clear the convergence rate is slower than plain SGD (never mind AdamW) by a factor proportional to the effective rank of the Hessian.