Hacker News Reader

I have two 3090s and it runs fine with `ollama run mixtral`. Although OP definitely meant mistral with the 7B note

ollama run mixtral will default to the quantized version (4bit IIRC). I'd guess this is why it can fit with two 3090s.

I'm using mixtral-8x7b-v0.1.Q4_K_M.gguf with llama.cpp and it only requires 25GB.

I have 128GB, but something is weird with Ollama. Even though for the Ollama Docker I only allow 90GB, it ends up using 128GB/128GB, so the system because very slow (mouse freezes).

What docker flags are you running?

None? The default ones from their docs.

The Docker also shows minimal usage for the ollama server which is also strange.

I run mixtral 6 bit quant very happily on my MacBook with 64 gb.

The smaller quants still require a 24gb card. 16 might work but doubt it

Sorry, it was from memory.

I have those models in Ollama:

I have those:

dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB)

The quantized one works fine on my 24GB 3090.