Glq a port trellis quantization of large language models as a vLLM plugin

glq is a re implementation gpl port of QTIP Trellis encoding of large language models as vLLM plugin. glq is efficient between 2-4 bits per word where it has an edge over scalar quantization. Fit larger LLMs on consumer hardware GPUs. glq extents QTIP up to 8 bits per word by running the original algorithm two times with a residual.

1.Incoherence processing — two-sided randomized Hadamard transform (via FWHT) makes weights ~i.i.d. Gaussian 2. BlockLDLQ rounding — sequential sweep with Hessian error feedback 3. Viterbi encoding — bitshift trellis + computed Gaussian code 4. Decode — stateless bit-shift arithmetic, fused into matmul https://github.com/cnygaard/glq/blob/main/docs/trellis-3inst...

glq comes with a user friendly installer which installs glq, vLLM and gradio chat interface and optionally a local coding agent. If you choose glq-chat glq sizes which model fits in VRAM downloads the model and starts a chat, on a 24GB Nvidia card Gemma4 is choosen. glq also ships with picode and then Qwen 3.8 27B is used.

glq also works on pure CPU. Because the LLM weights are compressed and are decompressed on the fly less DRAM/VRAM bandwidth is used.

The target group of glq is PC gamers with Linux and a Nvidia gaming card. If you happen to have an AMD Epyc 9005 server CPU the cpu encoding/decoding should be usable.

Original QTIP paper not affiliated but grateful to the original authors, https://arxiv.org/pdf/2406.11235