Hacker News Reader

Comment by antirez | Hacker News Reader

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".

View on HN

There is an issue for it:

https://github.com/ggerganov/llama.cpp/issues/4381

what a sick project to be able to attract a billionaire programmer [0] and c royalty.

[0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...

For those out of the loop, who are the billionaire programmer and C royalty people in this link?

Tobi is the founder of Shopify.

I also looked at the threads and had no idea who they were referencing.

Tobi Shopify and antirez (redis etc)

The linked comment is authored by the CEO of Shopify; the top-level comment in this thread is the .. what would you say, author/originator/founder of Redis. (And maybe/presumably more widely known in C-land for other projects or contributions, idk.)

I'm looking forward to a 4-bit quantized version that I can run on my Intel Thinkpad with 32GB RAM, should be almost usable if its as fast as a 14B size model.

Whether it's any good is another matter, I guess the leaderboard on HuggingFace will be updated at some point.

> so that it will be possible to do the inference in low ram doing something similar to "paging".

I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.cpp CPU inference).

The upside is that it should be as fast as running a single 7B model, which on llama.cpp should be in the ballpark of 10 tok/sec when 4-bit quantized.

For local usage, I'd say you will have an "OK" experience in a recent laptop with 64GB RAM, which is more realistic than having the necessary VRAM in a Nvidia GPU.

Correction, it should have the performance of a single 12B parameter model:

"Concretely, Mixtral has 45B total parameters but only uses 12B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12B model."

https://mistral.ai/news/mixtral-of-experts/

Reply on news.ycombinator.com