Sparse LLM Inference on CPU: 75% fewer parameters

huggingface.co

78 pointsfreediver2 comments

> The model was sparse fine-tuned on GSM8K, a dataset with high-quality and diverse grade school math problems. For this task zero-shot evaluation achieved 0% while 8-shot evaluation achieved 6.8%. The model was therefore fine-tuned via supervised fine-tuning (SFT).

That's a heck of a score to gloss over. For reference, the GSM8K dataset has questions like this:

> Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?

A LLM that can't get above 10% performance 8-shot on that is pretty much worthless by modern standards.

(Or at least worthless for math problems, but I haven't been very impressed by MPT in other areas either.)

So what the article tells us is that they managed to get garbage much faster than originally. Which .. yeah, that's not great.

That is the accuracy of the non-finetuned model i.e. mpt-7b. Once finetuned, it can get around 30% accuracy on 0-shot. Not sure what 8-shot performance is for the finetuned model. https://arxiv.org/pdf/2310.06927.pdf

Reply on news.ycombinator.com