There are lots ways to make LLMs more efficient:

- Pruning

- Distillation

- Sparse transformers

- Mixture or experts

- Quantizations

My understanding is that none of these are free and they all come with various trade offs.

For example MOE lets Mistral beat similarly sized models, and the inference performance stays close (only an incremental increase). But the training time is way more than a typical 7B model.

But which of these approaches gives the most bang for the buck?

Also consider it’s not either/or many of these techniques can be combined.

And maybe worst of all, some of the testing that can be done to find out doesn’t give the same answer with smaller/toy models.