And yet quadratic attention is still unavoidable. Anything that lets you have sub quadratic attention is going to have an accuracy Vs performance tradeoff.
No, you can make multiple passes across the data set. There's no reason that you have to use quadratic attention. I point to the shape of our own brains, which are limited in size. Yet, through multiple passes over information and random access, which we guide, we're able to process very large systems.