Top New Best Ask Show Jobs

atharv_jaju

4 karma · joined September 13, 2023

submissions comments

atharv_jaju··on Exllamav2: Inference library for running LLMs locally on consumer-class GPUs

I have 8GB RAM. So I am able to run 7B 4bit at maximum.

I used GGUF + llama.cpp.

Will try out Ollama.

Thanks for the info!

atharv_jaju··on Exllamav2: Inference library for running LLMs locally on consumer-class GPUs

Oh my bad, I meant how to code it?

atharv_jaju··on Exllamav2: Inference library for running LLMs locally on consumer-class GPUs

What is the best performing Codellama model on a Macbook with M1 chip?

Will Gguf 7Bn 4bit quantized model be good to run locally on the Mac?

atharv_jaju··on Exllamav2: Inference library for running LLMs locally on consumer-class GPUs

How do you count tokens per second?