Hacker News Reader

Comment by newfocogi | Hacker News Reader

Key claims:

"We have found three potential issues with Hoffmann et al.’s estimates of the Chinchilla scaling law that rely on Approach 3: 1. Their estimated model fits the reconstructed data very poorly. These conclusions hold even when accounting for potential noise in data reconstruction and excluding outlier models. 2. The confidence are implausibly tight given the number of data points. Obtaining confidence intervals that tight would require many hundreds of thousands of observations, while they likely had only ∼400. 3. Their estimated model implies a scaling policy that is inconsistent with their other approach"

Data point most people are probably looking for: "We find a range consistent with the 20 tokens per parameter rule of thumb. Indeed, our point estimates imply that 25.6 tokens per parameters is optimal."

View on HN

Their rule of thumb would imply that a 70B model is saturated with 1.7T tokens, that's inconsistent with reality.

The Chinchilla laws were compute optimal scaling laws. They're not supposed to tell you what parameter-token combination will saturate a model.

Compute optimal for what, training? There's nothing optimal in blowing up model size beyond the absolute minimal needed or you'll spent the equivalent of a country in electricity trying to scale inference later.

Yes, compute-optimal for training only. The purpose of the paper wasn’t to determine the most economically practical model one could build, the most “intelligent” model one could build given some amount of compute.

Quite. The big question at the time was "how much data do we need to train GPT-3 equivalent models". Open models had failed to live up to GPT performance, even ones with a massive number of parameters. So getting results that suggested a reason why other models were massively undertrained was important.

Meanwhile, people noticed that for deployed models, inference cost often outweighs the initial training costs. It's sometimes better to train a smaller, faster model longer on more data, because it has lower overall cost (including environmental impact) if you're expecting to run the model a few million or billion times (e.g., [1]). So training past the Chinchilla optimum point became a lot more common, particularly after Llama.

[1] https://arxiv.org/abs/2401.00448

Training yes.

Doubling your parameter count past that ratio will yield a better model than doubling your data and is much easier and cheaper to do.

That suggests that it's likely memorizing more special cases rather than distilling general principles. They generalize to some degree but clearly there's room for improvement.

It doesn't really suggest anything. Neither model will even close to saturation and all else equal, bigger models perform better in every way, including generalization.

But why do bigger models perform better? Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.

>But why do bigger models perform better?

No one really knows the answer to this question.

>Arguably because there's a larger state space that can be used to remember more contexts, which helps with both generalization and case-specific processing.

What I'm trying to say is that both models in either scenario are very over-parameterized and under-trained.

You say the answer is extra space ? The smaller model has not used anywhere near the space it has. They both have extra space.

It's like arguing a bigger drum is better because of extra space when all the water you plan to store will not take even half of your smaller drum

Blow up model size, get lots of space and parameters to do the double-descent grok thing in, then distill it way way down?

No their claim is that there are dimishing returns for a fixed compute budget (in training) to scaling up data past that threshold vs. scaling up params.

This doesn't take inference into account either, obviously.

Reply on news.ycombinator.com