Hacker News Reader

I've been unable to get Mixtral (mixtral-8x7b-instruct-v0.1.Q6_K.gguf) to run well on my M2 MacBook Air (24 GB). It's super slow and eventually freezes after about 12-15 tokens of a response. You should look at M3 options with more RAM -- 64 GB or even the weird-sounding 96 GB might be a good choice.

View on HN

https://www.reddit.com/r/LocalLLaMA/comments/17kcgjv/how_doe... reddit thread talks about some of the pros and cons of a m3 max with 128 Gb costing ~5-6K

You can buy multiple 4090s for that money and will get real GPUs including tensor cores. Still relevant it seems: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

Reply on news.ycombinator.com