Existential risk can be another big reason
74 karma · joined April 12, 2024
That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.
Getting diagnosed with ADHD and taking meds did the trick for me.
Not sure how relevant this is, since inability to focus was always an issue for me.
My demo uses 2 bit quantization to run llama3 models on any device with enough ram.
We live in a world with such companies, and we can still buy quality things. If there is a demand for the purely-human generated texts, they will be around. Perhaps a lot of people around you will read ai text instead, and you'll get upset because of it, but it's their choice. You'll still have your thing
volt-amperes are joules
1 GWh/year = (10 ** 9) / 24 / 365.25 / (10 ** 6) MW = 0.11 MW
70 GWh/year = 8 MW
1755 GWh/year = 200 MW
252 GWh/year = 29 MW`vllm.model_executor.layers.quantization.turboquant`
> The technique implemented here consists of the scalar case of the HIGGS quantization method (Malinovskii et al., "Pushing the Limits of Large Language Model Quantization via the Linearity Theorem", NAACL 2025; preprint arXiv:2411.17525): rotation + optimized grid + optional re-normalization, applied to KV cache compression. A first application of this approach to KV-cache compression is in "Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models" (Shutova et al., ICML 2025; preprint arXiv:2501.19392). Both these references pre-date the TurboQuant paper (Zandieh et al., ICLR 2026).
In-browser demo: https://galqiwi.github.io/aqlm-rs
Source code: https://github.com/galqiwi/demo-aqlm-rs