RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

Blog post: https://developers.googleblog.com/2024/04/gemma-family-expan...

Github: https://github.com/google-deepmind/recurrentgemma

HuggingFace: https://huggingface.co/google/recurrentgemma-2b

I'd be curious to see this architecture trained to a comparable level to Llama3.

8B params, 15T tokens of training.

Llama3 8B supercedes ChatGPT3.5. If RecurrentGemma scales, then it would be both faster and more capable than its Llama peer.

related previous discussion https://news.ycombinator.com/item?id=39993626