That isn't what it's doing and it's not what distillation is.
The smaller models are distillations, they use the same architecture they were using before.
The compute required for Llama-3.1-8B and DeepSeek-R1-Distill-Llama-8B are identical.
In general I agree that this is a rapidly advancing space, but specifically:
> the Llama 8B model trained on R1 outputs (DeepSeek-R1-Distill-Llama-8B), according to these benchmarks, is stronger than Claude 3.5 Sonnet
My point is that the words 'according to these benchmarks' is key here, because it's enormously unlikely (and this upheld by the reviews of people testing these distilled models), that:
> the Llama 8B model trained on R1 outputs (DeepSeek-R1-Distill-Llama-8B) is stronger than Claude 3.5 Sonnet
So, if you have two things:
1) Benchmark scores
2) A model that clearly is not actually that enormously better from the distillation process.
Clearly, clearly, one of those two things is wrong.
Either:
1) The benchmarks are meaningless.
2) People are somehow too stupid to be able to evalulate the 8B models and they really are as good as Claude sonnet.
...
Which of those seems more likely?
Perhaps I'm biased, or wrong, because I don't care about the benchmark scores, but my experience playing with these distilled models is that they're good, but they're not as good as sonnet; and that should come as absolutely no surprise to anyone.