Hacker News Reader

Comment by radium3d | Hacker News Reader

I believe NVIDIA's long term strategy will be to pivot from the data center to the public, and the public will utilize open models on NVIDIA hardware at home. This will come after the RAMpocalypse completes (when the new fabrication plants (China, Tesla/SpaceXAI/Intel) fully ramp up and start selling their RAM for cheap in the next few years). Data centers will be for training mostly.

View on HN

There’s a lot about this that would make sense.

But, not really at the current technologies. Kimi and GLM are fucking awesome, but I don’t have 3TB of VRAM to run them, and I don’t expect to even when ram prices drop.

So now you’re back to the scaling issue before talking about power and compute distribution.

Do you think you'll realistically need 3 TB RAM to run a sufficiently good model 1 to 2 years from now? I certainly don't. Considering what can already be done with 128 GB of relatively slow unified memory, imagine if efficiency improvements continue apace, the memory becomes 128 GB of HBM, the flash device becomes capable of sequential throughput matching today's DDR5, and such a system was affordable as a routine purchase for the average person.

The things in LLM “knows” are not infinite per its physical size. “Experts” don’t exist in a world where hardware capacity was a trivial issue.

in two years, yes I expect us to have much better models. Perhaps many times more efficient. But not 100 times.

Don't discount improvement in the ability of models to search for and ingest external data - either web or purpose built databases. I really don't think 3 TB will be necessary with the right model architecture and training environment.

GLM 5.3 runs like a charm in ~240G of VRAM (with half a million tokens context). Depending on what you've got that's between 3 and 10 GPUs. Fewer, larger GPUs will likely run it faster.

Yes you can run a quantized GLM on 240GB of ram.

Why do you think the public wants to self host models over using a cheaper solution hosted in the cloud?

Subscriptions are almost never actually cheaper.

It is cheaper to subscribe to AI for $500 a year for the rest of your life than to buy a machine capable of running a current frontier model with no subagents.

> current

Almost all the non-tech people I know acknowledge concerns about privacy but have no realistic alternative available to them. I think they won't want to self host until and unless it's easy to deploy, there's nothing to administrate, and the hardware cost is in the same ballpark as a new laptop was prior to the apocalypse.

Because the public prefers to use ad funded Google search over hosting a tiny web search index you can fit in memory. :)

Nigh impossible nowadays to expand now that sites are increasingly blocking indexing by unknown crawlers.

> Why do you think the public wants to self host models over using a cheaper solution hosted in the cloud?

you aren't truly in control of your compute in the cloud, and if you came to become reliant on ai, the cloud company could simply turn it off for you and you'd be shit outta luck. Unlike a regulated utility like electricity and water, where you cannot be discriminated against based on how you use said utility.

Therefore, having the ability to run local ai is a good thing. Currently, hardware constraints have kept it out of most hobbyist's hand, but i am hoping that will change soon enough (within 2-3 yrs).

The mega corps are looking to regulate ai such that individuals cannot run their private ai themselves, so that they can become more effective a monopoly. That's why it is crucial that ai is not regulated, but companies providing ai.

I do not believe such a level of control is sufficient to make the consumer market overtake the datacenter market. It's too niche.

Personally, the biggest reason is that if I'm investing in my business, and the business is going to rely on AI models, I need them hosted locally on my own hardware in a replicable way. The reason being is, if the AI subscription service changes the model, or drops support for it, or worse (goes out of business), you're done.

Reply on news.ycombinator.com