Hacker News Reader

You already can run a local llama instance on a high-end graphics card (6+ GB VRAM).

Yes, I can, but (see my edit) there's very little utility because the quality of output is very low.

Frankly anything worse than the ChatGPT-3.5 that runs on the "open"AI free demo isn't much of a tool.

And it's hilariously bad (in comparison to regular chatgpt).

And slow. They never tell you that quantization of many LLMs slows down your inference, sometimes by orders of magnitude.

It depends on the quantization method, but yes some of the most commonly used ones are extremely slow.