I won an ML lottery ticket, then squandered it
twitter.com
twitter.com
It's the first time in my career that I've seen evidence in the field that tiny ML models can outperform much larger ones, if you happen to win the initialization lottery.
FWIW, I was very careful to make sure all of the hyperparams and architecture were exactly the same, otherwise I'd suspect I was fooling myself.
It's a one-block autoencoder for RGB frames of 64x32 resolution.
Yesterday's run: loss 0.0002
Today's run: loss 0.0018
16-block run: loss 0.0012
It hurts knowing I had a chance to study this phenomenon and let it slip. But at least I can document it in case it's helpful to others.