Brainformers: Trading Simplicity for Efficiency

The Brainformer building block is designed using neural architecture search.

"Brainformer consistently outperforms the state-of-the-art dense and sparse Transformers, in terms of both quality and efficiency. A Brainformer model with 8 billion activated parameters per token demonstrates 2× faster training convergence and 5× faster step time compared to its GLaM counterpart."

Note that the 8B model mentioned above has 158B total parameters. The authors compare the training time to other sparsely activated models, but seemingly not to dense models.

I think parsimony is great for human readability but also creates a bias that excludes within reach solutions that overall have meaningfully better properties https://en.wikipedia.org/wiki/Evolved_antenna

Architecture search always works for improving efficiency, but it never beats a real algorithmic improvement (and there were lots of interesting ones being proposed for increasing context window / improving inference time in the last few months).