Just how much compute/memory are we saving here?
My understanding is that a 1BN transformer is about 2BN flops/inference, so about 1TFLOP for a 500 sequence of inferences (and also about several GB of memory)
What would be the equivalent RWKV (let ignore the inevitable loss penalty which could be significant..)