Nvidia's RTX 5000 Ada Now Available: AD102 with 32GB of GDDR6
tomshardware.com
tomshardware.com
If CUDA is 100% compliant on AMD we'll actually see competition in this space.
If a SWE salary is $200k, total cost is probably around $400k. A $4k GPU, if bought each year, increases SWE cost by 1-2%.
Buying a nice GPU was the best purchase my employer made, even if I use it every 6 months. It's already paid for itself many times over.
If employees are more efficient, I need fewer of them. People cost a lot of money relative to capital expenses like nice laptops, good chairs, or individual GPUs.
This kind of spending almost always pays off very quickly.
I'm all for investing in quality work experience. Decent equipment being the most obvious. Not only for the potential higher productivity, but also for its definite hr/retention effects. Investing in tangible production improvements and comfort is money spent 10x more effective than the hr semi-mandatory fun/social events that have metastasised throughout tech.
Where asking wasn't an outcome, in contrast to some other colleague that just went "I guess we will just deliver below potential then" I used to buy my own stuff because I respected my time, but it definitely didn't want to keep me working there.
Now this does not mean you have to shell out for every whim. If you want a 10.000€ PC setup instead of a 4.000€ one, I will need a better justification than 'custom loop and RGB'.
I can think of one specific use -- processing a data set -- which allowed me to do something automatically which would have used a few hours manual time. My hourly salary times hours saved on that one use already paid for the GPU.
I use Stable Diffusion many times to make images for presentations before it was mainstream (yes, there were a few months before everyone was using DALLE and SD). That had a noticeable impact which went well beyond the cost of the GPU.
And so on.
Yes, I could have done this at lower cost on cloud-based services with only a little bit more overhead, but:
1) I wouldn't have put in the extra work
2) If I did, I wouldn't have been able to justify the expense when things were at the half-baked stage
Things like Hugging Face are at a level where this DOES increase productivity by far more than it costs, and not in the abstract hr/retention effects way, but in terms of concrete output.
More critically, these tools are now part of our tool set. That means when new opportunities come up, we can leverage them.
Don't forget the scale at which these operate. $1000 extra in a GPU adds up quickly when you buy a thousand of them.
Perhaps in big tech it's different but in our huge multinational anything IT is seen as a cost center only and has to be on the cheap. Which a real focus on short term savings because who knows if this year's SVP will still be in that seat next year.
Nobody ever told me "let's take the most expensive option because it's the best in the long run" :(
1) Enterprise-grade design, with features like ECC. I like ECC.
2) VRAM is basically the limiting factor on running LLMs. On my GPUs, I look at cost versus VRAM. Fitting a model in one PC helps. Fitting a model on one GPU helps more.
3) It's just over half the power (250W versus 450W). The enterprise cards are clocked a little bit slower, which dramatically reduces power.
At least on the older Ampere cards, they also fit better. The consumer cards had giant cooling, which made sticking several in a computer more complex.
But you can (and should) undervolt RTX 4090s if you care about power efficiency. Anyone buying one would know how to.
https://betterprogramming.pub/limiting-your-gpu-power-consum...
Yes unfortunately I dont see 128GB VRAM coming any time soon even with 512bit GDDR Memory Interface.
I can buy a 32GB DIMM for <$50. Eight of them would do it for $400.
YES, I know the performance hit, but I'm not limited so much by performance as by which models fit. That's true of a lot of other people too.
Only way I can see this happening is if Intel or AMD get serious about AI. This would seriously undercut NVidia's business.
Nvidia uses RAM limitations as non linear cost scaling vector in the same way openAI used context window but that opens opportunity for competition to disrupt long term.
And while you're right, I personally have no interest in running two 4090s. My 24GB 3090s and RTX 5500s can feel a little meager at times, but 4090s are still overpriced and power hungry. If I'm moving up, it'll be for density, and a few hundred won't matter.
If you're using these for purely inference (which is what the 5000 seems like it's tuned for) the vram's the real bottleneck so you get a similar bang for your buck using last gen's cards vs. the bleeding edge.
I've also found the crypto crash has dumped a bunch of these well-worn cards on the market you can pick up a bit cheaper.
It's not a bad pick necessarily, but there's also a good reason the M2 Ultra isn't the inferencing chip of choice in the industry; nobody really supports CoreML or Metal acceleration.
What is the difference when you use llama.cpp or transformers for both? In my experience, the difference is pretty much far-and-away once you get the model loaded into RAM. My 3070 inferences about as fast as I've seen the M1 Ultra go (and my system is using a $500 chip, not a $5,000 one).
For the money, I think you're just not getting the most bang for your buck with Apple Silicon. It's fine if you want a Mac and use AI to justify the bigger one, but... it would be such a sore experience to spend $7,000 to avoid CUDA. It's the wrong walled-garden to pick if you care more about AI.
> (4 bit, I don't have enough RAM to run 8 bit on the M1).
That does make a big difference in inferencing speed, though. Lower precision models are much faster, if you ran 4bit on both machines it would probably close the gap significantly.