And this uses Hyena, that can be considered a "previous generation" of Mamba. I think that this anwsers the question about the scalability of SSM and the transformer finally found an opponent.
50% improvement on very large sequences is great. But it is not yet transformative (pun intended..)
Presumably FlashAttention already does lots of the less-than-quadratic improvement