Hacker News Reader

What quant and engine are you using? The smallest ones I’m seeing on ollama are > 100GB e.g. https://ollama.com/library/qwen3.8-flash-next/tags

This arch uses engrams aka per layer embeddings. Not all of those GBs need to go in vram.

Halogen flash server for strix halo with quant 4. Its offloading the PLE.