some catches I'm looking for: it remains to be seen if the generations are high quality (not just low perplexity) and if the architecture scales beyond 3B, but definitely super promising
See section 4 of the paper, 'Empirical Evaluation.' Meets or exceeds performance of similarly sized transformers on various benchmarks such as Hellaswag and Winogrande. Seems very promising.
Hi og_kalu! I agree. Let me add that the authors of RWKV and Linear Attention deserve some credit too, given the connection between those linear RNNs and these new "selective" SSM models. See my comment here: https://news.ycombinator.com/item?id=38529943