EagleX 1.7T: Soaring past LLaMA 7B 2T in both English and Multi-lang evals

To be clear, this is a 7B model. It's just trained on 1.7 trillion tokens. At first I was confused why they were making such a big deal of of a massive 1.7T model outperforming a 7B model.

By the way, GPT-4 has been predicted to be using a 1.7T model, although OpenAI has not confirmed this to my knowledge.

> All evals are 0 shot

My bet is that this is the reason they are scoring high in "their" benchmarks. For model which are just trained on completely unlabelled data like llama, 0 shot won't work well.

e.g. For llama Hellaswag accuracy is 57.13% in their benchmark compared to 78.59% in [1].

[1]: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

> [June 2024] v6 MoE model

How does that work for RWKV architecture? Wouldn't you have to feed the same data through all the experts regardless if they're currently active or not to keep the rolling state consistent? Or am I misunderstanding that architecture?

>Let’s look what went really badly: Math.

>We dug through the dataset we used for training, and realized we missed out the entire math dataset (along with a few others) due to an error. Oops.

This is kinda hilarious.

> All data shown here is made available in the Google Sheet over here:

Over where?

sounds dreamy. anyone know how I can install this on my m1 mac?