Hacker News Reader

Yep, the progress will be slow but inexorable on this front.

Sooner or later we'll arrive at what I see as the optimum point for "AI", which is when I can put an ATX case in my basement with a few GPUs in it and run my own private open source GPT-6 (or whatever), without needing to get into bed with the lesser of two ShitCos, (edit: and while deriving actual utility from the installation). That's the milestone that will really get my attention.

View on HN

You already can run a local llama instance on a high-end graphics card (6+ GB VRAM).

And it's hilariously bad (in comparison to regular chatgpt).

And slow. They never tell you that quantization of many LLMs slows down your inference, sometimes by orders of magnitude.

It depends on the quantization method, but yes some of the most commonly used ones are extremely slow.

Yes, I can, but (see my edit) there's very little utility because the quality of output is very low.

Frankly anything worse than the ChatGPT-3.5 that runs on the "open"AI free demo isn't much of a tool.

Reply on news.ycombinator.com