> The problem I have with this is that the claim that "inference has high gross margins" is paraded around as something that automatically makes all this make sense.
Except that this isn't the argument I'm making, and I don't think most of the people who are pointing it out are claiming that inference being sold at high gross margins means that you don't have to consider the massive investments that have been made.
The more common narrative is that these companies are losing massive amounts of money and will never be able to make a profit. And that's where looking at the investment in training versus the economics of inference is useful because, once again, inference actually does have high gross margins.
> R&D makes sense if we consider that research and development eventually gets to a model that doesn't need to be trained anymore, and can be served at relatively low cost indefinitely.
This just isn't true. If revenue grows at a sufficient enough rate versus training costs, and/or training costs decrease substantially enough, the numbers could work out over time.
People who are claiming that it's mathematically impossible are going on vibes. And just to be clear: I'm not claiming that it working out over time is guaranteed, or the likely outcome. But it is possible and the mechanics by which it would work out (if that comes to pass) is pretty simple.
> I agree that this is speculation. But it is based on actual sweetheart deals taking place (such as SpaceX not charging for compute during a period in Q2) and on Anthropic being very unclear if their compute cost is based on market rates or subsidized by providers who are also investors in the company.
Regarding SpaceX: my understanding is that as it was ramping up its capacity, which was not optimized and capable of being delivered in full at the time, it provided concessions, including discounts and deferred billing, to large customers. What SpaceX got from this in return was a massive backlog (over $14 billion worth) of non-cancellable contracts.
The concerns about Anthropic investors are legitimate, but again, you can only speculate. If your entire thesis ignores plausible scenarios that aren't so negative (like the possibility that compute is being sold at or near market rates), you have to be honest: the thesis is biased to support the argument you want to make.
If I was going to invest in any of these companies, I would want to know more about the terms of these deals but I don't think anyone who hasn't actually seen the agreements should make confident arguments based on what they think must be in them.
As a side note: if you really want to be exact, we'd need to know how compute purchased at below cost affects the income statement. If the below-market pricing was for compute related to training, it would not be the same on the income statement as compute related to inference. Whereas below-market pricing for inference compute would hit COGS and improve gross margins.
> You may have a major breakthrough that halves the compute use for a task. It is meaningless if the hardware is now 3 times more expensive.
What matters is the price per unit of compute. "The hardware is more expensive" only threatens viability if the price per unit of compute is rising, not just the price per chip.
You're throwing around random numbers to make a point but there is real data on this subject. For example, Epoch AI published a report a couple of months ago that found the cost of achieving a given level of AI performance has fallen about 47% per quarter since 2023 even as hardware and memory prices rose by tens of percent. On one of its science benchmarks the cost per question at a fixed score fell 725x in less than 18 months.
It did also find that large training runs have grown by 2-3x per year, which is the crux of everything, once again: training costs versus inference costs.