Hacker News Reader

Try a quantized model on llama.cpp — I have an M1 Max 32Gb and it’s running fairly fast in 20Gb of memory.

32GB memory only leaves about 24-26GB for the GPU by default which is quite low for a larger model like that. For comparison it runs great on a M2 Max 96GB.

It runs OK on my 32B Mac Mini, but I have mostly been running it using the inexpensive Mistral and AllScale APIs.

Do you need to use Mixtral? The typical quantized Mistral 7B and similar models can do summarization fine.

I'm using gpt-4 as an assistant for basic computing questions such helping with quick shell one liners or looking up for the simplest way to do some manipulation of rust values (vec of result into result of vec which I always forget how to do etc).

I wanted to try how local models compare with that use case of mine.

The only one that worked decently was Mixtral

Haha no not at all, but it’s fun!

Get a dirt cheap windows machine with >32GB or rent a VM?