Ask HN: What's the best hardware to run small/medium models locally?
M2 MacBook? A certain graphics card? Running inference on CPU on my old Thinkpad isn't fun.
1) What are you trying to do?
2) What's your budget?
Generically saying, "run inference" is like... you can do that on your current thinkpad, if you want a small enough model. If you want to run 7B or 13B or 34B models for document or sentiment analysis, or whatever, then you can move to the budget question.
When I was faced with this question, I bought the cheapest 4060 Ti with 16GB I could find. It does "okay". Here's an example run:
Llama.generate: prefix-match hit
llama_print_timings: load time = 627.53 ms
llama_print_timings: sample time = 415.30 ms / 200 runs ( 2.08 ms per token, 481.58 tokens per second)
llama_print_timings: prompt eval time = 162.12 ms / 62 tokens ( 2.61 ms per token, 382.44 tokens per second)
llama_print_timings: eval time = 8587.32 ms / 199 runs ( 43.15 ms per token, 23.17 tokens per second)
llama_print_timings: total time = 9498.89 ms
Output generated in 9.79 seconds (20.43 tokens/s, 200 tokens, context 63, seed 1836128893)
import os
import openai
url = "http://localhost:7860/v1"
openai_api_key = os.environ.get("OPENAI_API_KEY")
client = openai.OpenAI(base_url=url, api_key=openai_api_key)
result = client.chat.completions.create(
model="wizardlm_wizardcoder-python-13b-v1.0",
messages = [
{"role":"system", "content":"You are a helpful AI agent. You are honest and truthful"},
{"role":"user", "content": "What is the best approach when writing recursive functions?"},
]
print(result)
This is by far the most important question. Because frankly, I can run LLaMA on my raspberry pi. It's slow as hell and not suited for any real time task but there are definitely operations where this would be an appropriate cost effective solution (preferably with actually a smaller distilled model).
There is no one size-fits all solution. The general advice is going to be a general mid tier graphics card but I assume that's information OP already has or could have found just as easily by typing this question into Google or any LLM. So if you (OP) want better advice, we got to have more information. The more detailed, the better (if this is a commercial application, then the answer is A100 because geforce cards are not allowed to be used for commercial environments, but no one's really going to stop you either). Ask vague question, get vague answers. But we will ask refining questions to help you ask better questions too :)
If you want a laptop, that's going to be a very different machine to a desktop.
I did some investigating, and I found it doesn't start using the GPU unless you have a lot of input (such as a long prompt).
2. AMD: They may change the land scape in coming months. And it looks like the US gov restrictions on GPU's are going to impact price in the server market in 2024.
3. The stacks are evolving quickly. What you buy for today may be supersede by something tomorrow that means you should have spent more or could have spent less.
If you want to play: Ram, is what matters most. GPU ram and system ram (in that order). Get the best GPU you can (ram wise) under clock it and then add system memory if you can. Once you have a test bed that works for you, renting/cloud is a way to scale and play with bigger toys till you have a better sense of what you want and/or need.
Also as other have pointed out, it depends... I run models on Raspberry Pi's as well, one is doing live network detection...
Good GPUs and Apple hardware is pricey. Get a bit of automation setup with some cloud storage (e.g backblaze B2) and you can have a machine ready to run your personally fined tuned model rapidly with a CLI command or two.
There will be a break even point of course. Though a major advantage of renting is you can move easily as the tech does. You don't want to sink large amounts of money into a GPU only to find the next new hot open model needs more memory than you've got.
- https://vast.ai/ (linked by gchadwick above)
Personally I have 1x 4090 because I like gaming too, but it isn't really a big improvement over 3090 for ML unless you have a specific use for FP8, because VRAM capacity and bandwidth are very similar.
I can’t give this as a recommendation - there are far more tools available for Nvidia GPUs, but larger VRAM is available on AMD GPUs at lower prices from what I can see.
If not Mac, follow other advice with NVidia GPU. in term of the software ecosystem, NVidia >> Apple >> AMD > Intel. (I think I got the ordering right, but the magnitude of difference might be subjective.)
Of course with those you'll also have to spend some money on motherboard, ram, SSD, PSU, CPU, ect.
I think the best bang for the buck is probably a Mac studio with as much ram as you can afford.
I bought an RTX A2000 (12GB VRAM), and it's fine for 7B models and some 13B models with 4 bit quantization, but I kind of regret not getting something with more VRAM.
I don’t have any Mac experience.
Having their cards run in servers is a big part of their business model, and Linux owns that market, so I’m surprised if their support is getting worse.
Things like poor Wayland support—sure. But then, why would somebody use this matrix-multiplication accelerator to draw graphics, right?
I wonder if it is worth trying the iGPU, if you haven’t already.
It would be a shame to distro-hop for a non-preference-based reason like this.
You make a a good point, I've been having one card do double-duty as hdmi output and GPGPU. I'll try the motherboard's built-in hdmi and see how that goes.
I've been really busy with other stuff for a few weeks and haven't really thought about the best way to fix this. Thanks for the suggestion!
what are you talking about?
Also I run Linux full time, there's nothing better.
Apple side is still severely broken, you'll be fighting x86 shenanigans regularly.
Tested both Linux (some things will need manual patching) and windows. Works like a charm.
If you have twice the cash, go for a new RTX4090 for rougly twice the performance.
If you need more than 24GB vram, you want to get comfortable with sharding across a few of the 3090's, or spend a lott more on a 48, 80, 100 GB card.
If you feel adventurous, you can go a non nvidia route, but expect a lott of friction and elbow grease at least for now.
The 3060 is popular too, being a 3090 cut in half.
Calling the 3090 a "meme" makes it sound like the 3090 is a joke. Do you mean that the 3090 is "well-known" in local AI communities?
There are lots of questions about what hardware to get for ML, and the generic answer is basically always "get a 3090." Its so frequently recommended that it feels like a meme to me.
3060 RTX with 12GB VRAM if you're budget friendly, dial up from there.
Steer away from Apple unless all you do is work from a laptop.
If I had the cash I would go for 24GB M2/3 pro. That would allow me to comfortably load the 7B model in to ram.
Because there’s no CUDA, the speed is much slower than ChatGPT. The answers from 7B are also not at the same quality as ChatGPT. (Lots of mistakes and hallucinations)
It runs at under 3 tokens per second. I usually just give it my prompt and go make a coffee or something. The server is in my basement, you can barely hear the fans screaming at all.
Super easy to get setup and play around. Probably start with Llama2 13b or 7b, and then Mistral 7b.
If you want a dead-simple chat GUI for interfacing, check out Ollamac.
Kind of regretting not getting 64GB of ram now. I didn't think I'd need it, but... Here I am, wishing I could run some of these models.
I have 16GB too and have no trouble running even 13B models.
Edit: wizardcoder:7b-python-q4_1 throws the same error
I started playing with ComfyUI and Ollama.
An M1 studio ultra would generate a 'base' 512x512 image in around 6 seconds, and ollama responses seemed easily 'quick enough'. Faster than I could read.
On an I7-3930K, purely CPU only, a similar image would take around 2.5 minutes, and ollama was painful, as I would be waiting for the next word.
Then I switched to a 3080ti, which I hadn't been using for gaming as it got stupidly hot and I regretted having it. Suddenly it was redeemed.
On the 3080ti, the same images come out in less than a second, and ollama generation is even faster. Sure, I'm limited to 7B models for text (the mac could go much higher) and there will be limits with image size/complexity, but this thing is so much faster than I expected, and hardly generates any heat/noise at the same time - completely different to gaming. This is all a simple install under linux (pop os in this case).
tl;dr - A linux PC with a high-end GPU is the best value by far unless you really need big models, in my experience.