Hacker News Reader

Comment by ares623 | Hacker News Reader

Are they distilling from the big labs or Chinese labs at all? In that case isn't it $1 trillion frontier CAPEX + Inkling's costs for the total costs.

"I bought this house with my own savings plus a small million dollar loan from mum and dad."

View on HN

In their last technical report (for the original Inkling model), they mentioned 'To bootstrap post-training, we ran an initial SFT on synthetic data generated by open-weights models including Kimi K2.5. The bootstrap accounts for a small fraction of compute, with the majority being employed for large-scale RL on synthetic and human-created environments.' and in their most recent one they said 'The H3 and H4 mixtures consisted mostly of trajectories sampled from earlier models including Kimi K3 and our own models, filtered for correctness and alignment.' So it does seem like traces from larger models do contribute. I think Microsoft is one of the few companies that has actively made an effort to avoid distillation with their MAI series.

I think Mistral also avoids distilling on other company's models.

Everyone is building on everything that comes before.

Houses are not built from scratch either. The know-how, power, water, roads, materials…

Yes, and the sum total of those costs (that makes _some_ economic sense) end up on the total house price.

Is the same true for distillation?

If you’re going to do the cost math that way, be prepared to put a price on ~the total sum of all works created by humanity, which the frontier labs appropriated for their models.

Reply on news.ycombinator.com