Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.
There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).
At around 75% of the full model intelligence
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
But here my price per kwh is 40 cents (euro).
But if you integrated it with solar system and a heat exchange, this will be feasable
Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.
However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.