For the ones who are still wondering how so much profits, inference is a highly profitable domain: https://inferencex.semianalysis.com/profit-estimator
There is no reason to think inference can or will remain interestingly profitable even if you assume pure-play inference providers who don't do any model training. Serving an LLM isn't child's play but it's also not so hard as to count as a moat. Inferencing is:
1. Selling a largely undifferentiated product.
2. Into a highly competitive market.
3. To customers who are primarily sensitive to price.
Industries like this have always existed and in the modern era tend to have margins of about 3% to 5%. Any level of margin higher than that gets competed into the ground.
If you want to claim inference is highly profitable in general, you're going to have to answer why it won't end up like supermarkets or airlines, both of which have bigger moats than inference.
The other problem with this analysis is obviously that you can't just stop training models. Even if you don't care about improving capability or performance, the knowledge in their weights gets stale. Cost of inferencing has to take into account the ongoing cost of training, you can't cleanly split one from the other.
- I don't... "inference is profitable" is as of now. No one knows future, neither have I commented anything on it.
The statement is valid as it stands.
Because LLMs are like evolutionary crabs, they all get to the same point.
Boosters are right that AI is a historically impactful product, but big labs will be unable to meaningfully profit from ushering in the AI era just like Netscape failed despite ushering in the internet era.
Name a company with a currently high profit margin on inference. I can't.
OpenAI and Anthropic's valuations are as high as they are because investors hope that they will create AI that is much much more powerful than the AI they have now. The fact that they already have a substantial revenue stream helps, but it helps mainly by lending credibility to the hope that they might create something much much more powerful.
(You, Mike and I agree that selling the inference of models that are not much better than liberally-licensed open-weights models will be a low-margin business.)
It's not a huge problem for users but if margins are low it's a huge problem for the debt funded datacenter buildout. It's just pure finance at that point, about whether those GPUs can generate enough yield to beat deprecation and financing costs.
The big clouds generally don't sell compute at high margins. The margins differ a lot between services. Services like core VMs and object storage are usually low margin or even break even. Where they make the money is things where they've got proprietary software with competitive advantage, like managed scalable forks of Postgres, or egress charges that are there to create lockin. Managed DBs are pure SaaS and can generate SaaS like margins.
We're also starting to see efficiency gains, so we might meet halfway.
You can well imagine if you own your infra and can make utilization over 90%, where these go.
And then, my link above is for K3 model, which, compared to ds flash models, is highly unoptimized.
If I had a few million lying around, I would have invested in inference.
2. Newer gen nvidia gpu's like GB300 are way more profitable.