2023-09-13: Preliminary ROCm support added
Makes me curious how will the RTX4090/3090 compare with something 7900-ish
2023-09-13: Preliminary ROCm support added
Makes me curious how will the RTX4090/3090 compare with something 7900-ish
Here's how it compares on standardized llama2-7b 4K context testing vs some Nvidia cards: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
Not on the same platform so hard to make any solid conclusions, but vs the 4090 on a perf/$ basis right now, it's not too bad - about 60% of the inference performance of a 4090 at 45% the price. Still, if ML is your main use case, I think it's hard to argue that a used 3090 wouldn't be the way to go.
And I'm not in too much of a rush (weeks / short months). Can afford to wait for a month / two to see how the ROCm support pans out on the AMD side. 60% performance for <50% price is a compelling argument. If the ML support is decent...
Though the 4090 is somewhat expensive, I might end up taking it anyway just for the simple reason that it's the only 24GB card with CUDA.
"Expense" in my context means that I'll deduce about 50% off of the invoice come taxes time. I'm OK paying $400 of my own money for an AMD card, I'd think twice, but probably accept coughing up $800 for a 4090, but $2k is definitely over the budget at this stage.
Thanks for sharing the spreadsheet with Your benchmarks though. You're going into my "HN commenters to watch" list ;-)
If you're just dipping your toe in I'd recommend using Google Colab or cloud rentals like Runpod/Vast.ai which will be more cost effective if you're not running 24/7 or want to have access immediately on tap.
Thanks for the rental tips though. I'll take a look. It's indeed more effective in the beginning...
Not for 70B. You can finetune 7B or 13B.
Wouldn't that, in theory, be what consumer LLM users want? A mid to low range GPU with tons of VRAM?
Or am I missing something here with regards to memory bandwidth? (sort of started this discussion a while back on the localllama lemmy: https://lemmy.ca/post/3910263?scrollToComments=true )