I was inclined to listen to your take… then you started talking about having 6 graphics cards in your computer and acting like that’s a normal thing that people do…
This 6 GPU setup will probably outspend OpenRouter on electricity alone.
If all those nvidia gpus the hyperscalers bought were online the cost to rent a single b200 wouldn't be $50 and hour and you could buy those 6 year old gpus I use for $200 each. Not almost $2k a pop they sold for now.
A 180gb b200 on runpod is $6.8/hr. Nowhere near $50/hr.
The _most expensive_ on-demand b200 rental I can find is $11.2/hr. There's a number of websites around these days that track GPU rental prices across different services: gpus.io, gpu.watchworks.dev (vast and runpod only), gputracker.net (not free, lol), gpurentalprices.com, priceofcompute.com, rentgpu.org, etc (I'm only looking at the first page of search results for GPU rental prices tracker search). Most of these don't include vast or runpod, but all of them show prices under $10, mostly around $6/hr for b200.
I signed up with half a dozen providers that had such numbers on their main page. When I tried to actually rent one, every time no such instances were available or they were more like 5x what you mention at the cheapest.
Ergo 6 GPUs isn't a typical setup, never has been and most likely never will.
OP is a classic outlier dev who is able and willing to set up and maintain such a thing - most normal people aren't.
That's what, the same as a Mac studio?
I do not enjoy the fact those cards cost more than their msrp 6 years later, but there is a much more important consideration than money (which also makes sense, but about that later). It is the fact soon people will not be able to do my job without AI at all. Even now if I didn't use it I think I'd be out competed very quickly.
And having the ability to run it locally, using a really useful, not toy model is very useful. It makes you independent from Anthropic deciding to ban your account for example.
As for money, it is an open secret the biggest cost of coding agents use is input tokens not generation. I tend to use about 1.3B input tokens per week on claude code with only 7-8M out. Out if this 80% is cached. And the cache is pretty restrictive. You have 5min cache and 1h cache. If you don't keep reading over that time your cache expires on the cloud. Then your 500k context counts as 500k input in its entirety.
And the numbers I mentioned would cost thousands of USD a week at API prices.
But when you control inference, you can keep your cache for as long as you want and save it to disk.
I tend to have up to 10 coding agent sessions open at a time. Some are used once a week. I never use more than 5 at the same moment. Having 15 full contexts cached in RAM basically moves my local cache utilisation to 95%+
Basically I think the AI companies will soon require us to pay the real price for the inference. I prefer to be ready.