Hacker News Reader

Comment by colingauvin | Hacker News Reader

Where is the actual evidence of distillation? I keep seeing this repeated ad nauseam but I must have somehow missed the evidence.

View on HN

Distillation a pretty well documented technique that actually pre-dates LLMs https://arxiv.org/pdf/1503.02531

Here is a project that guides you through it if you want to prove to yourself that it works https://github.com/arcee-ai/DistillKit

I took GP as asking for evidence that the reduced-cost Sol is actually a distillation of the previous-cost Sol. AFAIK, providers distilling or quantising models and offering them as the same model have not been proven.

I doubt he was claiming that. He's probably saying that the ability of Chinese companies to be able to distill frontier US models has put downwards pressure on the price of all models.

I'm saying that it is unclear that without distillation this wouldn't still be happening. There is a massive narrative that no one but OpenAI, Anthropic, and Google can make a model without distilling. But there's basically no evidence of that.

I think the (non-definitive) evidence is that the chinese models are always just slightly behind the publically released US models, and never ahead.

I think it's more that distilling is enormously cheaper than training from scratch to achieve the same results

Alternatively, modern AI is good enough at optimizing its own kernels that it just keeps pushing costs down. Unlike the semi-decentralized inference provider community, OpenAI has both the talent and the compute to throw at the problem of making their models much more efficient to run.

GPU kernel optimization is just the kind of well-bounded problem with clear success criteria that AI loves.

That distillation exists isn't the question.

It's about evidence this is an active force in competition in LLMs.

I think the biggest actual piece of evidence is how hard the major players are trying to stop it

To be precise, the distillation mentioned in this paper is not the distillation used by other model companies. In the one mentioned in the paper, your teacher and student model typically have similar architectures - and you typically need access to the full logits. What happens here instead is motivated by the fact that these companies don't have access to the training data and compute that anthropic/openai have. The distillation they do basically amounts to using traces from ant/oai models trained on much more data with a lot more compute (in many cases including the hidden intermediate tokens! turns out there were many ways to coax it out) and then either directly training on it or using it in many ways in post training pipelines. It falls under imitation learning, IMO.

It turns out you can train a 1b model at almost 1000 tokens/s on a m5 max laptop. As a personal experiment, I've been asking Sol for synthetic training data and synthetic agentic training data (model distillation in it's purest form), plus modified opencode, codex transcripts etc for training data, and nobody's even paying me to do it. If I'm doing it has a hobby, you can bet industrial users are doing it.

Been using a lot of Kimi K3 lately and the answers have been… „load-bearing“ to the point of hilariousness. It‘s obvious from where they distilled, even if sceptics rightly point out it can‘t have been the only source of their secret sauce, as it‘s been better than the current Opus 4.x at the time of release.

Qwen 3.8 27b also enjoys their genuinely load-bearing seams.

Why does Kimi insist its name is Claude?

Please elaborate the mechanisms by which a LLM would know what model it is.

ask it what type of model it is or what it's name is... it's weird that Kimi will say it's claude...

Okay, but I would also ask “why does Claude say that its name is Claude?”

Because it's in the system prompt

AFAIK, you cannot run Claude without one of their mandatory system prompts at all.

If we wanted to compare model responses, we would give all models system prompts with model names, thereby fixing the Kimi misattribution.

The reason Kimi often states its name as Claude is likely because we can actually run it without the mandatory system prompt, smoothing over awkward competitor mentions.

The training data likely references Claude significantly more often than Kimi, given the popularity of the models. There will simply be more examples of “Claude” being the response to that question.

Doesn't Claude say its Deepseek when asked in Chinese? I remember there being posts about that a while ago.

Unless it is specifically instructed in the system prompt it will give you the most likely answer, which is Claude. If there are instructions in the system prompt it will give the correct answer, which would be Kimi. Or are you suggesting that Kimi is copy pasting Claudes system prompt?

By being trained on text containing "I am X" in the model response section.

Musk confirmed in federal court that xAI does it: https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x...

It's also how providers build their smaller models out of their larger ones; they publicly talk about the process.

The evidence is Anthropic's own reporting [1]. You may doubt that they're telling the truth, but that's what they're reporting.

[1] https://www.anthropic.com/news/detecting-and-preventing-dist...

"Distillation attacks", and it's just the exact same thing they did to the rest of the web and all human works.

Explain “exact same thing”? Like it load the Wikipedia page for Fort Worth, Texas, does it make Wikipedia spin up an editor to go write the article?

I’m really not sure how you’re getting “exact” here.

Anthropic scraped the whole web, scanned every book, pirated every bit of media to feed in to their training. Distillers are doing essentially the same thing scraping all the knowledge from the LLM to create a training set for a new one.

They are crying about theft after committing the largest theft in human history.

Why is it theft to read all content ever

If it's not theft to do that, then it isn't theft to distill the models. Anthropic wants it both ways.

I'm supposed to believe that DeepSeek distilled a 300B-1T param model with 150,000 requests? Lol.

Are you thinking distillation goes from zero to complete model?

I believe it can be used in the RL / fine tuning sense, in which case 150,000 requests, assuming every one was detected, could move the needle in quality.

I agree it couldn’t replace all of pre and post training , but I don’t think that’s the claim. You do typical training, then distill really difficult cases.

Here's an example: https://github.com/microsoft/Build25-LAB329

there is no evidence. it shortcuts post training by a huge margin this is true. but that is all.

https://techcrunch.com/2026/04/30/elon-musk-testifies-that-x...

Make sure to stay updated!

Nothing against what I said in there. Distillation is far from enough to get to the frontier. Its at best a ramp (the most efficient one used by everyone ) that shortcut and saves millions of rl runs before a model moves.

You said there isn't evidence of distillation. Yes there is

I said there is no evidence distillation is all you need as per the initial comment "The fact that AI models can be so easily distilled and replicated is such a stroke of luck." it is far from trivial to reproduce the capabilities of anthropic models for instance

And probably saves a lot of energy which would be good for the planet

"No evidence" -> Large lab saying that they use distillation when training their models

the initial post infers that distillation is all you need. it is not. in 2026 you need large scale distributed inference of gigantic models, rl envs and millions of dollars runs to get to something decent. if you think glm just has to sft on traces of claude to edge Mythos on some cyber benchmark you are fooled.

Reply on news.ycombinator.com