I'm looking for hardware recommendations for running LLMs locally. Specifically, best bang for bucks at the following price points: $2,000 $5,000 $10,000
What more could you do with a 10k budget than with a 5k budget?
Thank you
The most important part to run LLMs might be the VRAM-size. Since modern Macs use unified memory you simplified get RAM=VRAM, so a 64GB MacBook Pro might outperform an NVidia 4090 in LLMs.
However the specialized cards from nvidia have more VRAM and are generally faster.
The LLM of your choice should fit into vram, so if you only need 21gb, one 4090 will do the job
On a different computer running Cinnamon Mint I used GPT4ALL to load a Llama 3 8b model. I use it to read a bunch of local docs for me. It also didn't cost anything.
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama run llama3.2:1b
Then try bigger models if that is too fast for you :)
https://github.com/Jeffser/Alpaca
GPT4ALL on the cinnamon mint box I first installed Easy Flatpak then installed it from there. That I had a little different goal, I had a lot of pdf and epub books I wanted to index. I downloaded a small model again, a 4.34G Llama 3 8B Instruct. In the local docs I put one pdf 'Github for Dummies' and asked it to index. It took about 30 minutes IIRC. (don't start with too many, it could be days indexing. You can add docs later and re-index so you can build your database slowly) Then instead of looking up in the book how to revert a commit, you chat with the model and ask it how to revert a commit.
My take is the model is the human language interface for queries, and that's all. Bear in mind almost everyone knows more about this me, I just wanted to see what all the fuss was about.
You can also do similar what I did with gpt4all online for free as long as you have a google account. Go to notebooklm dot google dot com and start uploading your documents.
SomeOddCodeGuy in Reddit shared his very cool multi LLM setup [1]. I quote:
> I have 2 "assistants" set up, with the intention to later add a third. Each assistant is essentially built to be an advanced "rubber duck" (as in the rubber duck programming method where you talk through a problem to an inanimate object and it helps you solve this problem). Each assistant is built entirely to talk through problems with me, of any kind, and help me solve them by challenging me, answering my questions, or using a specific set of instructions on how to think through issues in unique ways. Each assistant is built to be different, and thus solve things differently.
Each assistant is made up of multiple LLMs. Some examples would be:
A responder model, which does the talking
A RAG model, which I use for pulling data from the offline wikipedia api for factual questions
A reasoning model, for thinking through a response before the responder answers
A coding model, for handle code issues and math issues.
See his post at https://www.reddit.com/r/LocalLLaMA/comments/1i1kz1c/sharing...