In fact, I just switched from a 96 GB ram laptop to a 32 GB ram laptop specifically so I can have GPU acceleration because 20 toks can’t replace the 6000toks or whatever astra does
Not today. But when it turns into a $600 brick you plug in next to your router, that's gonna work for most people.
For today, I'm speaking to software developers - who can 100% handle setting up a local AI system - it's only as hard as building a PC.
I asked an AI agent to setup local AI inference for me. I gave it my use cases. It downloaded all the required bits, installed and configured them. Benchmarked different settings and models for my hardware and my use cases and then produced a nice report with findings and recommendations.
It is significantly easier than building a PC.
A quick gut check would be to look at the model release date. If it's older than the training cutoff for whatever you used, that's probably not the best thing you could be running.