Oh yes? I'd be intrigued to learn more.. I am working with smaller sequences (peptides, len is about 30ish) and currently use an attention based mechanism, do you maybe have pointers to where to start?
You are likely to find kernels for Gated Delta Net 2 as Nvidia had a paper on that.
https://news.ycombinator.com/threads?id=ActorNightly&next=49...
TLDR its not so much about the transformer, as much as expanding dimentionality. You can chose any which way to do so.