Thank you. The OP is from the authors of Hyena, and I agree it's unclear if it will scale as well as self-attention in transformers. The paper on RNNs looks very interesting. I like the fact that it keeps things simple and familiar. I've added it to my reading list.