In a related subject, what’s the best hardware to run local LLM’s for this use case? Assuming a budget of no more of $2.5K.
And, is there an open source implementation of an agentic workflow (search tools and others) to use it with local LLM’s?
And, is there an open source implementation of an agentic workflow (search tools and others) to use it with local LLM’s?
Also none of this is worth the money because it's simply not possible to run the same kinds of models you pay for online on a standard home system. Things like ChatGPT 4o use more VRAM than you'll ever be able to scrounge up unless your budget is closer to $10,000-25,000+. Think multiple RTX A6000 cards or similar. So ultimately you're better off just paying for the online hosted services
Of course the economics are completely at odds with any real engineering: nobody wants you to use smaller local models, nobody wants you to consider cost/efficiency saving
This is more of a social problem. Read through r/LocalLlama every so often and you'll see how people are optimizing their usage.
Seems like there would be cost advantages and always-online advantages. And the risk of a desktop computer getting damaged/stolen is much lower than for laptops.