some catches I'm looking for: it remains to be seen if the generations are high quality (not just low perplexity) and if the architecture scales beyond 3B, but definitely super promising
See section 4 of the paper, 'Empirical Evaluation.' Meets or exceeds performance of similarly sized transformers on various benchmarks such as Hellaswag and Winogrande. Seems very promising.