Comment by ToucanLoucan | Hacker News Reader

Comment by ToucanLoucan | Hacker News Reader

Parent Full thread

> Cost tends to go down with time as compute becomes cheaper.

This is generally true but seems to be, if anything, inverted for AI. These models cost billions to train in compute, and OpenAI thus far has needed to put out a brand new one roughly annually in order to stay relevant. This would be akin to Apple putting out a new iPhone that costed billions to engineer year over year, but was giving the things away for free on the corner and only asking for money for the versions with more storage and what have you.

The vast majority of AI adjacent companies too are just repackaging OpenAI's LLMs, the exceptions being ones like Meta, which certainly has a more solid basis what with being tied to an incredibly profitable product in Facebook, but also... it's Meta and I'm sure as shit not using their AI for anything, because it's Meta.

I did some back of napkin math in a comment a ways back and landed on that in order to break even merely on training costs, not including the rest of the expenditure of the company, they would need to charge all of their current subscribers $150 per month, up from... I think the most expensive right now is about $20? So nearly an 8 fold price increase, with no attrition, to again break even. And I'm guessing all these investors they've had are not interested in a 0 sum.

This reasoning about the subscription price etc is undermined by the actual prices OpenAI are charging -

The price of a model capable of 4o mini level performance used to be 100x higher.

Yes, literally 100x. The original "davinci model" (and I paid $5 figures for using it throughout 2021-2022) cost $0.06/1k tokens.

So it's not inverting in running costs (which are the thing that will kill a company). Struggling with training costs (which is where you correctly identify OpenAI is spending) will stop growth perhaps, but won't kill you if you have to pull the plug.

I suspect subscription prices are based on market capture and perceived customer value, plus plans for training, not running costs.

> So it's not inverting in running costs (which are the thing that will kill a company). Struggling with training costs (which is where you correctly identify OpenAI is spending) will stop growth perhaps, but won't kill you if you have to pull the plug.

I don’t think it’s that cut and dried though. Many users run into similar issues as other issues with things like reasoning (which is (allegedly) being addressed) and hallucinations (less so) both of which in turn become core reasons for subsequent better versions of the tech. Whether the subsequent versions deliver on those promises is irrelevant (though they often don’t) to that, at least IMHO, being a core reason to “stay on board” with the product. I have to think if they announced tomorrow they couldn’t afford to train the next one that there would be a pretty substantial attrition of paying users, which then makes it even harder to resume training in the future, no?

The closest analog seems to be bitcoin mining, which continually increases difficulty. And if you've ever researched how many bitcoin miners go under...

It's nothing like bitcoin mining. Bitcoin mining is intentionally designed so that it gets harder as people use it more, no matter what.

With LLMs, if you have a use case which can run on an H100 or whatever and costs $4/hour, and the LLM has acceptable performance, it's going to be cheaper in a couple years.

Now, all these companies are improving their models but they're doing that in search of magical new applications the $4/hour model I'm using today can't do. If the $4/hour model works today, you don't have to worry about the cost going up. It will work at the same price or cheaper in the future.

But OpenAI has to keep releasing new ever-increasing models to justify it all. There is a reason they are talking about nuclear reactors and Sam needing 7 trillion dollars.

One other difference from Bitcoin is that the price of Bitcoin rises to make it all worth it, but we have the opposite expectation with AI where users will eventually need to pay much more than now to use it, but people only use it now because it is free or heavily subsidized. I agree that current models are pretty good and the price of those may go down with time but that should be even more concerning to OpenAI.

> But OpenAI has to keep releasing new ever-increasing models to justify it all.

There seems to be some renewed interest for smaller, possibly better-designed LLMs. I don’t know if this really lowers training costs, but it makes inference cheaper. I suspect at some point we’ll have clusters of smaller models, possibly activated when needed like in MoE LLMs, rather than ever-increasing humongous models with 3T parameters.

Reply on news.ycombinator.com