Hacker News Reader

Comment by NitpickLawyer | Hacker News Reader

> but you can achieve the same at 1/10th of the cost.

For some tasks, sure. But not for all tasks. And for some tasks, cost per token is irrelevant if it provides real benefits that are oom compared to what you had.

Local models are indeed becoming "good enough" for some tasks, but there are still tasks that they can't touch. There's a recent benchmark for kernel writing. Fable wrote a kernel that provides ~30% more throughput per unit of compute compared to the latest Opus max / gpt max. Does it matter how much that session cost in terms of one session if you can take that kernel, deploy it on your inference fleet and "magically" get 30% more tokens served to your clients? There are companies that would pay millions for such a "leap". Because they can make more millions down the line.

View on HN

You're looking at the status quo and ignoring the trajectory. The best current open models are about as good as closed models from ~1.5 generations ago. The rate of improvement of all models is converging to zero. It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.

The problem is going to become that there's no incentive for anyone to run the stupidly-expensive training phase. May God have mercy on the stock market.

> The rate of improvement of all models is converging to zero.

That's so obviously not true that I don't even think it's worth the energy to even debate it. It's been said for years, yet here we are, constantly improving. People really don't get RL / the bitter lesson, do they?

> It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.

Not a chance. There's hundreds of billions of dollars on one side, and oom less on the other. There's also scaling laws and information theory. No matter how good, a 30B model will not be able to be better than a 3T+ model, all things being equal.

You are mistaking models becoming "good enough" for an increasingly number of tasks, which I agree is happening, with SotA models stagnating, hitting walls etc. That will not happen for many many years to come.

>The rate of improvement of all models is converging to zero.

Curious where you draw this conclusion from? Most benchmarks still show continual steady progress https://metr.org/time-horizons/

To me it seems like a first-year physics scaling laws problem. To get linear improvements in capability, you appear to need need exponential (or at least superlinear) increases in model size. We have no technical nor business solution for that kind of scaling, so the long-term outcome is obvious.

The question is: what proportion of tasks can not be handled by GLM5.2?

How many software developers were working on code like the one you describe?

On that aspect, I agree. Smaller / open models are becoming "good enough" at an increasing number of tasks. And that's great for us, consumers. But there will always be tasks that are "worth" pursuing with better models, and cost is irrelevant for those tasks. That was the point I was trying to make.

That’s true, there will always be demand for ultra-intelligent assistants, especially if they surpass what humans can achieve at similar cost. For the other 90%, the average frontier model will be good enough.

I don't disagree with you, but it's important to pay attention to where the money is. Cheap non frontier models is something that Anthropic and open AI could do too, but who's willing to pay a premium for using them? It will be like competing to sell rice, lots of demand at Rock bottom margins.

You're saying it like it would be a problem.

Reply on news.ycombinator.com