Hacker News Reader

Why is there a surprise when a larger model works better? And call it emergent?

There's no a priori reason to assume that models would see a step function improvement in abilities as their size increases. Perhaps the null hypothesis would be that we'd observe a linear increase in capability. Empirical evidence shows that's not the case, and that's why studying this phenomena is interesting.

The judgement criteria itself looks step-like for many of these things.

Word unscrambling, for instance: does a 15 percentage point increase in accuracy suggest that you're 15x better at it (from close to 0 to 15%ish)? Or could you do that by being modestly better, but that's just enough to cross the threshold to solve the simplest 15% of words in the task challenge you're being graded on?

We're also using a log scale on the X axis there but a linear one for the Y axis which amplifies the "steppiness"

This is a fair point. Things like arithmetic either work or they don't. Is partial credit issued as a linear interpolation between the predicted value and expected value? If not, then you necessarily will have step-like behavior in the data.

The surprise is that it's qualitatively better. Like being able to talk and follow instructions where smaller models couldn't do either at all.

Reply on news.ycombinator.com