Hacker News Reader

Comment by tehjoker | Hacker News Reader

At least for now, those keys haven't been completely unlocked, so parameter scaling is the primary way corporations are achieving increases in ability. If they hit the hardware ceiling, maybe the research direction changes then.

I forget who said it, but a trenchant insight is that current neural networks are generalized function approximators and essentially they are doing through brute force what biology accomplishes elegantly. Our hardware (both in the sense of our brains in terms of elegance and our computer chips in terms of raw throughput) is amazing.

View on HN

> so parameter scaling is the primary way corporations are achieving increases in ability

Thinking tokens have nothing to do with parameter count and where some of the biggest gains have come from.

We're really past the point of scaling parameters being viable. We simply lack the content to make "bigger" better at this point.

No, parameter scaling still works because it broadly increases sample efficiency. We don't have the raw data to make a Chinchilla optimal 100T, but that doesn't mean there are no gains in building up to 100T.

I don't think "biology" and "elegantly" belongs in the same sentence. Let alone the one you wrote.

I suspect that the "elegant" workings of the human brain are a pile of hacks. Evolution crammed the brain full of patterns that are a good fit for the specific problems it typically solves - the way curve detectors and scale invariance are a good fit for CNNs.

We don't have millions of years of evolution of non-random init, but what we have is enormous datasets and compute at scale.

The brain is a pile of hacks, but it's still the most advanced thinking machine on the planet and it works on barely any power.

LLMs are very competitive on power use at inference time.

Sure, frontier LLMs require timeslots on massive server arrays. But they also get things done a lot faster than humans do. The net favors the LLM.

Reply on news.ycombinator.com