Hacker News Reader

Comment by nightski | Hacker News Reader

At that cost I'd just buy some GPUs and run a local model though. Maybe a couple RTX 6000s.

That's about as much as my Framework Desktop cost (thankful that I bought it before all the supply craziness we're seeing across the industry). In the relatively small amount of time I've spent tinkering with it, I've used a local LLM to do some real tasks. It's not as powerful as Claude, but given the immaturity in the local LLM space—on both the hardware and software side—I think it has real potential.

Cloud services have a head-start for quite a few reasons, but I really think we could see local LLMs coming into their own over the next 3-5 years.

Same but I imagine once prices start rising the prices of GPUs that can run any decent local models will soar (again) as well. You and I wouldn’t be the only person with this idea right?

I mean, will it? I would expect that all those GPUs and servers will ends up somewhere. Look on old Xeon servers, it all ended up in China. Nobody sane will buy 1U serve home, but Chinese has recycled these servers by making X99 motherboards which takes RAMs and Xeon CPUs from these noise servers and turning into PCs.

I would expect that they could sell something like AI computers with lot of GPU power created from similar recycled GPU clusters ussed today.

Those won’t be sufficient to run SOTA/trillion parameter models

And most tasks don't demand that.

Distilled models are good enough.

Reply on news.ycombinator.com