Hacker News Reader

Comment by sandworm101 | Hacker News Reader

It is part of the trend to locally-hosted AI. I demoed my home rig (3x5060s) to my boss last week. It certainly was not as performant as commercial options, but was well above the 90% solution for what we needed. The boss was shocked at the price points for home-built hardware. Gaming GPUs are pricy, but are nowhere near the prices for commercial (rack-mount) compute hardware.

(People actually came by my desk to see "the AI", the physical thing. It blows people's minds to realize AI doesn't need a data center, that it is just some computer parts sitting on a desk.)

The big hitter AI companies see the writing on the wall: open source models and local AI rigs will decimate the market for big AI in the next few years.

View on HN

I see this as the eventually end game. Open source models and local hardware are going to be the only path forward for companies as the AI providers move to a constant need to increase prices to drive profitability and stock price.

Just like on-prem killed cloud?

There's no particular benefit to keeping the hardware local unless you have extraordinary security requirements. Over time most of the workloads will end up in data centers with the cheapest electricity and real estate.

Can you post a picture? It sounds super cool. Was it a specialized hardware, or a consumer mobo with 3 PCIe slots?

It is just a Asrock motherboard and 1000w corsair power supply sitting on the bottom shelf of a "mining rack" open air case. The GPUs are attached with janky pci risers (the motherboard has four pci slots). I went with 5060s so I could stay with "short" 2-fan cards. The boot drive is a 500GB NVME with an encrypted 2tb NVME for storage, both on the mobo. I added four cheap fans and an oldschool (with a nob) fan controller to circulate air away from the GPUs. RAM options are maxed out at 4x16GB DDR4 sticks. Expansion options include adding a fourth 5060 and any number of SATA storage drives for content/models.

It, of course, runs on linux. I went with stock ubuntu so I could attach a screen and/or remote in to a full GUI. Having a full GUI also makes the rig more relatable to people who's eye glaze over when they see a terminal interface. "It's just a funny-looking desktop computer."

At 48GB of VRAM it can run basically any LLM I would want to download. Speeds are fine for a single user (Tok/s numbers change based on model/mode). But if I were using it for something like ComyfUI I would ditch the multiple 5060s and go with a 5070ti or 5080 as ComfyUI does better with a single powerful GPU.

It's not locally hosted, as they say they will be accessing the same models but through a controlled system rather than on their own accounts

Interesting! How many token/s using what model are you getting out of it? And what are the other specs?

Cool, but that sounds out of reach financially and technically for most people

The article is about the Swedish Government, hopefully it wouldn't be out of reach for them.

I don't think it will be the moment nvidia decide there's more money to be made in selling hardware for local-first inference. They'll do the training themselves and give the models away for free, like the toy in a box of cornflakes.

I'm sure cloud models will always have a slight edge but most tasks, most of the time, won't need that few extra percentage points of performance.

In a lot of sci-fi games an upgrade path for ship/cybernetic hardware is to replace the 'AI chip'. If the models performance ever start to plateau I can't wait to see AI models being baked into the chip itself.

Somebody did it... here we go: https://taalas.com/the-path-to-ubiquitous-ai/

Llama 3.1 8B baked into an ASIC. Expect 1/10th the power requirements, and other savings.

This is one of the reasons I don't believe the "it'll never be profitable" narrative. Just stop training when you think your model's "good enough" and move to custom silicon.

>> out of reach financially and technically for most people

Yes and no. I bought most of the hardware before the rampocalypse. It is just an AM4 chip on a nothing motherboard. Even today, 16GB 5060s are maybe 500 each. 64GB of DDR4 is another 500$. Add a power supply and pci risers and the basic parts could be purchased today for around 3000$... call it 4000$ all-in including my labor to get everything connected and talking to each other.

4000$ is money but is also almost nothing in the world of commercial IT purchases.

They're using Copilot, not locally-hosted AI.

Reply on news.ycombinator.com