Something is changing in the unit economics of software
nicolo.xyz
nicolo.xyz
Right now the "hosting" cost for inference is per-unit because it's new and expensive, but that won't last.
There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app.
The bigger problem for SaaS is that the floor has risen - people can build their own solutions for things that they used to buy SaaS for. So the industry needs to level up and solve harder problems.
It’s so cheap that companies choose to spend more on AI inference (more reasoning, more capabilities, longer context), not less - see Jevons paradox.
I don't know about "very fast" or "soon" unless you're speaking in geological terms.
SOTA models like Kimi 3 require thousands of GB of RAM/VRAM to run at speeds that are real-time useful. Manufacturing the memory necessary for that quantity to be available at app-hosting prices will take decades. Software efficiency solutions might drop needed memory by an order of magnitude in that time...but a tenth of an enormous amount is still pretty darn big so won't get us there "soon".
They claim to support 24/7 agent coding with no VC subsidizations or money lost on a subscription, instead relying on optimizations on agent selection. I think it'll shift sooner rather than later.
Take Qwen3.6 or such, figure out how many users you can batch-share one server between. server_cost_per_month / users_per_server = monthly_cost.
Apparently, H200s are $1.57/h with commitments or <$2/h with spot pricing, so server_cost_per_month=1200 maybe (not my area). batch_throughput=3400tps apparently, assume 15% utilization of the actual LLM (building, testing, human typing, idle, etc; remember, no concurrency offered, "1 line") at SLO 100tps you could pack about users_per_server=227 for $5.28/user/month cost. Easy to build rate limiting to throttle users to a fixed max speed. At that point providing the actual VM where the code runs is a comparable cost.
Actual measurements may vary, but batching is the gimmick.
(Whether Qwen3.6 is good enough to pay for is kinda beside the point; they're specifically avoiding saying what model it is, so don't expect you're getting full-speed SOTA 24/7. You don't know whether they're selling you Qwen3.6 or what. They're also apparently forcing you into their custom harness, system prompts, virtual machines... very little here says anything about whether it works well or not. Also no specs about the VM they put you on. The fact that they say "VPS" scares me.)
The article have talked about this:
> You cannot, however, build a business model today on a cost breakthrough that has not happened yet. Even if costs do fall, Jevons paradox[5] tends to kick in: historically, when a resource gets cheaper, we find more uses for it rather than consuming less. Cheaper inference does not get banked as margin improvement; it gets consumed by deeper integration, more calls per interaction, more autonomous agents running in the background. The cost per call drops but the calls per user multiply.
Are there any examples of products containing ai inference that are successful? Products that are beyond just direct access to frontier LLM's, I mean.
Products are only viable when you do a lot of work to solve a problem, which is then shared by many potential customers. It makes sense to amortize the high costs of solving the problem across all the different customers to reap economies of scale. Businesses then pay for product design, hosting, and maintenance because those costs can also be amortized, and they are cheaper than a bespoke solution for each customer. But if the bespoke solution becomes cheaper than that, because it's generated by an LLM that doesn't need to be paid a living wage, there's no reason to have the product in the first place. Just solve your damn problem and let other people solve theirs.
This is an underrated factor in the market structure of the AI bubble going on now. Anecdotally, we're not seeing new AI-based products other than the foundational models and coding assistants gain traction. Why? Because AI makes it so easy to customize the solution to your particular needs that everybody is just solving their particular needs directly. It's the opposite of the Internet boom, where the network greatly expanded the potential market, reduced the cost of reaching them, and made it economical to spend large amounts of money building a software product that had a TAM of billions. The AI boom instead enables extremely cheap customization, which shrinks the market to a single customer who uses AI to directly solve their problem rather than building a product that's generally applicable.
How am I going to use ChatGPT instead of, say, Figma or an office suite with my coworkers?? It's no use for it to spin up a clone for me if my coworkers can't collaborate. And if they can, what it will serve is a product.
How can it be cheaper for every company to re-invent the wheel?
> Products are only viable when you do a lot of work to solve a problem, which is then shared by many potential customers.
Yes, that's what businesses pay services for.
Companies not matching their prices to current reality, mostly.
Is it as good as a professional could've done with Photoshop? No. But it's about 90-95% of the way there, which is good enough that neither her nor any customers would care.
A great deal of business is precisely this, tasks that need to be done but where you only care about "good enough" solutions. After all, a key principle of business is you don't outsource your competitive advantages. You pay people for the table stakes, the things that everybody needs but that they only need to be "good enough". LLMs can generate "good enough" facsimiles of a wide variety of fields.
Remember that in the mid-00s, this is what YouTube and the social web was. Folks doing little, productive economic activity, usually unpaid, which can now reach an audience through technology. And now the top career aspiration in young kids is "Be a social media influencer." That's how disruptive innovations start: with little transactions that aren't considered business at all, which then grow until they subsume the economy that everybody knows of.
Whether this could happen is a critical question.
What ive found currently is that its not possible because the non-determinism of the model means you cant trust it, and these tasks are such that people expect determinism. Even if they dont then they dont tolerate the kind of stupid mistakes these models make
What could happen is that the model becomes good enough to be near deterministic, or as deterministic as humans. I dont know if the current next token prediction foundation is sufficient for that. But who knows, maybe it is
For a lot of tasks, classical deterministic computer programming is better than an LLM. It's just that LLMs are very good at classical deterministic computer programming now.
About design and the product side of maintainance, what i have felt is that, deep down inside, the user does know exactly what they need. If they can learn to communicate that in a way another person or agent can understand, then design and maintainance will be fine. Thats a big if, but if the economics align for people to benefit from developing that skill, maybe they do.
This is the part I'm least convinced by.
Corporates used to develop their own tools and they moved away from it for many reasons. Cost of development was only one of those reasons.
Not really.
>Every inference call costs money.
Not really, either. If you buy your own GPU, rack it, and run an open model, there is no unit cost. This is just expensive hosting infra. You also pay unit costs for SaaS that your software uses (things like SMS etc).
No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.
Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.
So the only way it’s a fixed cost is if you don’t pay for power. If you only consider the cost of the power, it might still be cheaper paying for an api.
Spoken like a guy who has never had to maintain bare metal infrastructure ops at scale. These things break, need re-imaged, have parts that break, have to be configured (now you need provisioning pipelines, monitoring, alarms, etc), have to be maintained when something goes wrong (swapping hardware and software, debugging the alarms into actually figuring out which bits are broken and/or misconfigured), have to be catalogued, have to be planned for, deprecated, and the finances accounted for through complicated accounting to show investors the Capex at quarterly meetings.
Then you have to make fleet decisions on how much of each type of server you will want to buy, expanded storage, how long you will support each generation of server, when you will order new hardware, how to order new hardware, lag for real world installs, hiring actual humans to fly around to all of your datacenters and do the actual installations / maintainance / etc. Then you will have to do contracts with individual datacenter operators for margins, electricity rates, hosting contracts, white glove ops hourly rates, etc.
Businesses that own their own hardware tend to have a lot of employees whose jobs are maintaining it and running the business side of that.
We need a better word for this because the things you didn't do are not a cost
TFA contends that there is some fundamental shift in the economics of software, but it doesn't look to be very different from either a new SaaS dependency or racking new hardware.
common misconception about software unit economics. With enterprise software (one that costs real $$$) there always more costs attached post shipping. Before client/server it was support. Then it was security and the constant threat of cyber attacks.
Distributing software is nothing like distributing books.
Absolutely not. Customers want systems for sales, reservations, accounting, and taking stock. That's where almost all the SaaS money is and none of it benefits from AI - and never will.
But, it's a hard problem. The models that run locally on normal computers/phones are pretty terrible compared to the frontier, without specialization and fine-tuning. And, even with specialization and fine-tuning, often a high-end general purpose model is going to do a better job and people don't need a bunch of local tools installed to do their various tasks.
I think this is the critical point that would be interesting to see if it holds. Technology seemingly tends towards increased specialization.
The bitter lesson says the exact opposite.
It's not all that different from saying "Don't optimize software, just wait for faster hardware". Yet highly optimized software exist, and performance on currently available stacks is a competitive advantage.
What gives? I say: Staying ahead of the "Don't optimize, just wait" curve can absolutely make sense. At worst, your advantage decays after a few years. At best, you stay ahead by n number of years and keep increasing the gap as you invest more.
If you make a specialist model and harness that takes six months to train and launch, will you have enough time to recoup your costs and make a profit before the generalists get good enough and cheap enough in your area of specialty and become the simpler option?
We already see "skills" repos containing a bunch of specialist stuff being pushed to general models via prose instructions for how to do various things and use various tools; and those models will do the tasks much more expensively and slowly, but without needing the user to find/buy/learn something specific to do the job.
The bitter lesson may be narrow, but the fact is people have been using spreadsheets for everything for 50 years, and only sometimes can a specialist tool effectively displace it in an organization. Very powerful generalist models are starting to feel like spreadsheets: The default way to solve problems in organizations. You might be able to pick off some of the specialist cases, but you might also miss.
Perhaps we can call this "The sweet dollars lesson".
Most LLM-using mobile games are already at incredibly weak TTP scores and perhaps the only current stratum of mobile games where TTP is almost always ahead of TTFA. TTFA before TTP is almost nonexistent because they claim to need a monthly subscription as soon as immediately after install, despite being advertised as free to start. It's also one of the few types of games where the paywall explicitly does not include "no ads". Some of these games running on monthly subscriptions still need ads for unit costs.
(The bulk of mobile games try for a sweet spot of TTFA in the order of hours of gameplay and TTP in the order of days of gameplay. Easier to get people hooked on your game if you can give them a few hours of uninterrupted fun up front.)
It seems pretty condemning of software economics with LLMs involved.
Of course, and so does everything in the software world. The point is getting the cost so low that it’s basically free. The new DS V4 Flash or the smaller Qwen3.6 models are still really expensive compared to what we were used to in the economics of software, but it’s not unreasonable to expect these costs to continue falling down.
Rough chatgpt estimate says 3-5 orders of magnitude of difference compared to a typical user interaction with a SPA (db/cache lookup, CDN…)
Well, no. Copying is free, or so near free it makes zero sense to charge. LLMs are just papering over the damage caused by profit.
Inference will be always more expensive than db operations or copying, sure. But how much more expensive is the question.
In this scenario the user would either sign into the platform via their Anthropic / OpenAI / Gemini account or use their API keys, and any of their usage would be billed to them directly.
The company then doesn't have to worry about the increasing costs from the AI usage.
Of course, in this future, AI providers become the new "Facebooks" of the world.
There are so many videos with hooks/teasers/'you won't believe what we discovered!!1', and now I just pause the video in the first second, ask "what's the tldr" and get the value from the video without a single ad impression (and likely racking up far more opex for Youtube than if I just streamed the video).
How do we make subscribers become equally comfortable paying for AI usage? Tokens, credits, inference calls?
Most SaaS already works this way. M365 or Adobe Creative Cloud are great examples. They value it like a life insurance policy and find ways to make you sticky. It’s easier to just buy it.
The first round of AI products suck because they are not well defined. Copilot only makes sense if you do shit in office and SharePoint isn’t a dumpster fire. In my large O365 environment the bottom 50% of users use less storage than the top 2%. So why would i buy copilot for my janitor?
When M365 E9 reconciles invoices automatically with Excel, I’ll pay $150/mo and fire a bunch of people.
Much as people may not want to like it, "software" as a product to buy and sell, even as a subscription, is probably going away, and will make about as much sense as "math" as a product.
We were already headed in this direction, but AI's going to rapidly accelerate this.