Hacker News Reader

Comment by jmspring | Hacker News Reader

The work going on to improve things on single card like a 3090 is nuts. HyperQwen, other things. I have my subscriptions, but always good to explore options.

View on HN

You have to understand that Strata and HyperQwen are giving you good looking tokens per second but are not giving you anything close to the bf16 model’s capability (they quantize, worse they quantize KV caches, they approximate expert routing etc.)

The models are so impressive that getting 70-80% of their capability can be helpful, but beware, you are using a significantly dumber model.

I have subscriptions for necessary work and even pay for api calls.

The machine with the 3090 I built years ago and suddenly it’s worth 4x plus what I paid. It is for testing, but I use the model I run on it and the model I run on an mbp M1 Max with 64gig as workers for specifically scoped work.

I’d love to have a few dgx sparcs or the like to do heavy local stuff, but that money is best paying the pimps for now.

With luna's pricing, is the power consumed not likely to more than the cost of API calls?

At 50tps on a 3090 for qwen 28b or similar, that's about 200k tokens/hour. Which is what...10c for Luna (balanced input vs output). 1h of power at 30c/kwh is 30c? So you get a worse model, for more cost?

I pay about $0.14/kwh and don't need it all the time. API calls might be cheaper. I develop in a way where I break things out where well defined tasks can be handed to a local model.

Things like the hadamard rotations being applied before the quantisation mean that it has much less impact than before.

Thanks for the hardmard reference. It's pretty amazing what people can do with maths. I should have studied that instead of software.

If you look at the open weights official checkpoints they are either fp8 or even 4 (kimi). None is using bf16 for inference.

Reply on news.ycombinator.com