Hacker News Reader

Comment by achrono | Hacker News Reader

Yes, evidence is needed but especially for the claim that distillation is what makes these open models good. Serious citation needed.

Think about it: even if they distill the shit out of frontier models, the model still gotta learn, right?

If anything, as you can see from the K2 Horizon release, aggressive (self-proclaimed) reliance on distillation does not result in a model that has remotely any frontier capability. Try asking K2 Horizon to write iambic pentameter for instance, or even give it the car wash prompt. I tried both these on the Q8 quant for the 7B model and the results were depressing.

View on HN

To whoever downvoted this, it would be helpful if you actually reply with something substantive. The post I'm responding to makes sweeping characterizations, and I challenge it with a relatively good heuristic and indirect evidence, and only get downvoted?

Because it's evident you didn't engage with the last sentence I wrote properly. "Citation needed" in more words is not a sufficient response.

> As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.

To help you further understand, the Chinese labs will never admit they were distilling until they are better than US labs with their own non-distilling process or some sort of espionage-like public reveal shows it. So you need to look at secondary indicators. Much like how the chinese government lies about their economic stats so 3rd parties use secondary indicators to figure it out.

My point is simply this: Chinese labs going to great lengths to distill Claude is evidence that distillation is useful but not that that is what makes them so good. That one doesn't even need them to admit it.

Nathan Lambert has made this distinction explicitly: he thinks Chinese labs likely innovate heavily on distillation, while saying he wouldn't call it a crucial factor in their post-training capabilities, in part because the RL work still has to be done by the lab itself -- just like what I said above.

https://www.interconnects.ai/p/how-much-does-distillation-re...

Reply on news.ycombinator.com