Anyway, with FIR you typically need many, many times the coefficients to get similar filter cutoff performance as a what few IIR coefficients can do.
You can convert a IIR to a FIR using for example the window design method[3], where if you use a rectangular window function you essentially unroll the recursion but stop after some finite depth.
Similarly it seems unrolling the TRM you end up with the traditional LLM architecture of many repeated attention+ff blocks, minus the global feedback part. And unlike a true IIR, the TRM does implement a finite cut-off, so in that sense is more like a traditional FIR/LLM than the structure suggest.
So, would perhaps be interesting to compare the TRM network to a similarly unrolled version.
Then again, maybe this is all mad ramblings from a sleep deprived mind.
[1]: https://en.wikipedia.org/wiki/Finite_impulse_response
[2]: https://en.wikipedia.org/wiki/Infinite_impulse_response
[3]: https://en.wikipedia.org/wiki/Finite_impulse_response#Window...