Other approaches for long sequence lengths:
- "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf
- "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf
It's unclear (at least to me) if any of these scale as well as transformers at this point.