Hacker News Reader

Comment by zer00eyz | Hacker News Reader

> Current top models are mere (mere in big datacenter terms) trillions of parameters right now. ... We'll soon start hitting 10's of trillions and 100's of trillions of parameter models.

Chinchilla Law says no, we wont. We're likely well past the point of diminishing returns but MOE hides a fair bit of it.

We already know that power laws apply to scaling so 10x the size is going to result in minor gains - where costs become a real factor (as if they weren't already).

FTA "The core philosophy there is scaling, and aggressively not research."

This is where the real gains for AI come from - good enough and CHEAP is going to win the day here (brute force, mixture of models). It's the engineers displacing the academics - and that will mean a bloodbath for revenue (see DeepInfer for what the end game looks like).

View on HN

I'm not an AI booster (in a "should" not "can" sense), but in terms of "can" we are going to see next generation hardware (i.e. Vera Rubin GPU architecture) and improvements in model density (i.e. loosely improvements in intelligence per millions of weights), so the models will get bigger, the models will get more dense, and we may see more "expert models" i.e. in the sense that for a while there were human generalists that knew all of science and mathematics until the 1800s, after a certain point we had to specialize since there was too much to know. The models will need to partially specialize after hitting the hardware ceiling.

Bioinspired algorithms may eventually lead to vastly improved intelligence per a weight too.

I don't think we should do this, but I feel doomed in that I suspect that we "can".

> The models will need to partially specialize after hitting the hardware ceiling.

I'm incredibly skeptical of anything that goes "oh, we need to split AI into specialist systems".

I suspect that a lot of the delta between "our best models" and "base models they're trained on" lies in relatively low amounts of parameter changes driven by SFT and RL.

Model capacity is not the limit - the limit is our ability to drive those changes efficiently. Even things like "good research taste" could be represented as tiny in relation to total model capacity, if they could be trained into a model efficiently.

At least for now, those keys haven't been completely unlocked, so parameter scaling is the primary way corporations are achieving increases in ability. If they hit the hardware ceiling, maybe the research direction changes then.

I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.

> so parameter scaling is the primary way corporations are achieving increases in ability

Thinking tokens have nothing to do with parameter count and where some of the biggest gains have come from.

We're really past the point of scaling parameters being viable. We simply lack the content to make "bigger" better at this point.

Reply on news.ycombinator.com