Hacker News Reader

Comment by akmarinov | Hacker News Reader

You sure can, https://github.com/lxe/dsx gets you this deepseek on 2x3090s and 128 gb ram at about 30 t/s

At around 75% of the full model intelligence

Hello it is I, lxe

Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.

However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.

Nice, thank you for your work, I contributed with a PR, to make it work on my CPU

About half the t/s of an online provider, heat, noise, power consumption and on top of that you only get 75% of the intelligence? Sounds like a waste of time to me.

As luck would have it, you can just pay the provider and use their service. Others may want to run locally for privacy reasons or just for the fun of it. I think at-home AI is incredibly awesome and I applaud the enthusiasts pushing it forward. Just because that's where it's at today doesn't mean it's the best we can do.

Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.

I use only providers because I don't have the hardware, but if, and that's a big if, the only moat left for humans is our ability to steer and make bigger picture decisions with our personality, I think it makes sense not to want to send all that conversation data to a 3rd party.

Half the token/s - sure, no way around beating big ass data centers.

Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.

75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.

It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.

I have not capped my 2x4090s so my room gets cosy warm.

But here my price per kwh is 40 cents (euro).

But if you integrated it with solar system and a heat exchange, this will be feasable

I specifically tuned it for maximum heat and noise

I mean it’ll be pretty nice come winter

Reply on news.ycombinator.com