None of that matters when there is literally a 100x multiplier on efficiency because of batching. It's all a rounding error.
You do not get to exclude the power grid costs. That is the core of the grift. They are trying to get data centers connected to the grid as fast as possible with contracted kwh rates so everyone else on the grid would end up paying higher rates. They can contract for existing wind power at 2 cents a kwh, then other businesses and residential customers paying for the peaker plants and grid upgrades.
You don't have to build a single data center or build power plants to keep gpus at home.
The 100x multiplier is complete nonsense. There is no truth to it. They are still running power through chips on the same nm scale. It is not saving anything. A gpu in a home only needs to run when you have it do something, so its much more efficient. OpenAI doesn't even use its current data centers, that is why they have internal teams using lots of gpu resources to front run other people's accomplishments. Those gpus are otherwise sitting idle.
You cannot claim a rented llm for $200 a month is cheaper and more efficient than a $1300 gpu you buy once and use for the cost of electricity.
You leave out the cost to the consumer, the fact that the consumer still needs home internet, electricity, and a computer to use cloud AI. Your "efficient" service didn't eliminate anything and is using the same amount of power to process the same amount of bits.
> They are still running power through chips on the same nm scale. It is not saving anything.
What even is your argument here? I explained why the centralized inference gets more throughput out of the same power. Let's use a real example, with Qwen3-32B (4bit quantized). Yes, it's old, but that won't change anything, it's just what I happen to have numbers for.
A home system with a RTX 5080 GPU can run it at ~45tok/s for ~300W, or ~7J per token. A DGX B300 can run it at 200000tok/s (split into 50-75tok/s streams) for ~17kW, or, 0.09J per token.
Not quite 100x, but getting there. The centralized system fundamentally needs to do less work total for the same amount of tokens, because it can reuse the load of model parameters for multiple streams.
> You cannot claim a rented llm for $200 a month is cheaper and more efficient than a $1300 gpu you buy once and use for the cost of electricity.
I'm claiming no such thing, don't rent a LLM for $200 a month, at least unless you are spending a lot more tokens than that $1300 GPU can do. There are options that are serving at market-determined prices, in addition to the hilariously over-priced options that the frontier labs provide.
> You leave out the cost to the consumer, the fact that the consumer still needs home internet, electricity, and a computer to use cloud AI. Your "efficient" service didn't eliminate anything and is using the same amount of power to process the same amount of bits.
You do understand that the consumer uses those whether they do their inference themself or not, and that their computer in fact uses a lot more power when doing inference vs when it is not doing inference?
Your fundamental failure here seems to be inability to multiply. The data center is not serving a single consumer. If you took the hardware, and distributed it to every home, you would need ~100x the power to provide the same amount of inference. Because the data center is in fact serving a lot of consumers at once.
And I like local inference. I want to control my own digital future. But wanting that and pretending that it is somehow more efficient are two very different things. People like us can acquire the hardware needed to run a local LLM, because we are hobbyists who are willing to spend money inefficiently for a hobby, but if everyone did it, it would mean that we would collectively need about ~100x more hardware and consume about ~100x more power.