Hacker News Reader

Hi there, 8bit and 4bit is currently supported on main. GPTQ is working in progress, as well as GGML

GPTQ support would be amazing (AutoGPTQ is an easy way to integrate GPTQ support - it's basically just importing autogptq and switching out 1 line in the model loading code).

Reply on news.ycombinator.com