Hacker News Reader

Comment by giwook | Hacker News Reader

I think the performance of these models on consumer hardware have a long way to go before abliterated* models have the potential to go mainstream.

View on HN

Damn spell check. I typed it correctly the first time.

Anyway, I have 8 r9700 GPUs racked up that can run a dozen or so sessions of qwen 3.8 flash next at once at > 100tps each.

I build operating systems and fix compilers with them and reverse engineer anything I want. Never given money to Sam or Dario or anyone else for tokens. If consumer gpus and models never got any better, I am all set.

So in this case it may not be that people haven't "figured out" that this is possible but that they don't have $20k lying around to experiment with.

But yet they can afford to pay Anthropic $200/mo forever?

This hardware pays for itself pretty fast.

Prices will of course come WAY down, but even today, even rideshare and delivery drivers still manage to get loans to buy cars that cost tens of thousands of dollars because they need it do their job.

People that need AI to do their job should consider owning the means of production as a top priority IMO instead of being perpetual renters without any privacy or defense against getting priced out.

> This hardware pays for itself pretty fast.

I think that's only true for a fairly limited set of people, though. Just the basic hardware in your case is about 80 months worth of the $200/month subscription. And that's excluding the machine to put the hardware in and the electricity, space, and cooling requirements.

If I'm someone who lives in a small apartment, the idea of running a bunch of 300W TDP cards is potentially problematic beyond just the basic cost.

"8 r9700 GPUs" is not a consumer-level system.

Not mass consumer, but pretty well within the reach of mass affluent, who regularly spend more on e.g. a fancier car.

True, but "within the reach of" and "would actually drop $20k to buy" are two very different things.

Those are all pcie cards marketed and sold to consumers that run on consumer boards. I bought one at a time over months whenever I find cheap used ones on ebay, always under 2k. A single card would be plenty if this was not my career and if I was not working on many things in parallel.

This whole rig cost less than the car the typical fast food worker bought with a loan to get to work.

This is awesome and needs to be known as The Way.

The average consumer might buy one of them. Not 8.

I very much want this to be true, I would shift as much of my workload as possible to local models. But is qwen 3.8 really good enough to do reverse engineering work? Is it because you have many sessions running at once, so it can "brute force" its way through problems?

Many sessions working in parallel breaking a problem into small parts can chew through any complex task I have been able to imagine. Even complex software reproducibility and compiler bugs that previously I spent months solving as one of something like a dozen people in the world with substantial expertise in this area.

I literally put 3 GPUs in a machine and let it crank for 2 weeks until it built a microkernel OS for itself and wrote its own network driver to get online a few days ago.

I am not sure many more complex problems in software exist than that.

There is no reason for me to ever subscribe to any AI providers other than maybe code-review diversity from different models.

How much guidance and initial thinking do you have to give it to solve complex problems like that? I've gotten spoiled with the state of the art models, where I give it extremely high-level goals "get this game playable in this other game's engine", it asks me a bunch of clarifying questions, plans it all, and steadily burns tokens on it until it's done. If I could get that locally with < $20K of hardware I'd be very tempted.

Reply on news.ycombinator.com