At what point do you buy a Mac studio or AMD Gorgon Halo and try running some of these tasks in local models? Surely don’t need frontier models for all this work.
And, in fact, I got it from £2500 to £150 by careful model selection (I ran a trial over a small dataset with different models until I found the least expensive/capable model that would reliably do what I needed), using a local model for one step, and paring down the inputs as well so the token count plummeted.
They feel like they're super smart but it's just as smart as hiring a $500k/yr PhD as a PA to schedule your meetings and answer the phone. Sure it does a good job but you're just wasting money and abilities.
They feel super edgy and bleeding edge but all they really do differently is switching the AI chat from auto to the most expensive option all the time. When I see what they're using it for they're only grasping the basics of its abilities.
They're also people sending 2 page AI generated emails that make no sense for literally everything where a "Yeah sure let's do that" would have sufficed.
Unfortunately most of these are business leaders who just want to look edgy and "all in on AI" so we can't really limit their tokens. Yet they're also the ones saying we spend too much and want to limit some of our dev teams actually putting it to good use.
Management succumbed to AI psychosis, and so we were directed to show that we were adopting and using AI, but what we were using it for was not considered important. So we went out of our way to burn as many tokens as possible.
If you tell Claude to be inefficient and burn as many tokens as it can, it’ll happily oblige. It will also offer useful advice as to how to use even more tokens.
Once you factor in cost of electricity and depreciation of hardware, it is more expensive to run locally.
There are many arguments for running local but cost isn't one of them.
Let's do quick back of the napkin math.
- Mac Studio M5 Ultra, 80-core GPU, 256GB memory, 2TB storage -- $11,299.00.
- Power (385w continually 100% of month) is $40/mo (my regional) or $480 annually. Let's say power is 3x more expensive in California, which it probably is, that's still only $1,440 annually in power.