How Nvidia’s CUDA Monopoly in Machine Learning Is Breaking | Hacker News Reader

How Nvidia’s CUDA Monopoly in Machine Learning Is Breaking

semianalysis.com

166 pointspella71 comments

I wish this were so, but I'm very, very sceptical. Everything I use is built to CUDA. Nothing actually works on non-NVidia hardware. "Actually works" is different from "theoretically works." A lot of what I use theoretically supports ROCm, but in practice, when you try to use it, you have enough bugs big and small, and things crash or don't work right.

That means everything I build is locked to NVidia as well, and everything people build on top of my stuff will be locked to NVidia too. Trying to maintain compatibility is like making Wine run Windows apps natively.

The lock-in is now very significant. Virtually no one else has shipped anything really stable and usable, and we're now a very long time into people coding and testing only on NVidia.

I hear that higher-level APIs are being built, nominally intended to be cross-platform, but I suspect (1) those will go the same way (2) we'll want backwards-compatibility. I have code which uses old ML models, CuPy, or otherwise. That stuff is locked to NVidia. I'm buying NVidia. This means I'm only making patches to bugs impacting NVidia. Network effects.

So long as most people buy NVidia, NVidia will have more engineering resources too. It's very hard for a smaller player to catch up or pass. See Intel for many decades.

To break from NVidia would require either a gross failure in execution by NVidia or something fundamentally new. To break the monopoly, I could see new products which might deliver:

- Very low latency

- Support for conventional DDR in large quantity (so I can run GPT-3 models at home)

- Much more general functionality (e.g. an open architecture, and integration with conventional Python to parallelize normal code)

... or other significant new features. However, I don't see a lot of stuff like that being worked on, let alone coming out.

AMD has a really weird view on how the programming world works internally. Some years ago they had some ARM server chips. Did they allow people access? No. They only allowed it for "Serious customers" and the whole thing failed as no software was available for it.

Same applies to their compute stuff. They really really want you to purchase their Pro cards and whatnot, and if you have a consumer card well too bad. Go back to playing games. Whereas with my Nvidia card CUDA runs flawlessly which leads me to doing test projects at home and fixing a bug here and there.

AMD doesn't understand how much of development happens on either home computers or university labs with small budgets which means consumer cards. They seem to think that everything happens using big teams and big money. Even though in reality it's the small developers doing stuff that then drives the purchase of the very lucrative large clusters.

For all of it's downsides NVidia actually understand the grassroots approach. Which is why their stuff just works on consumer stuff too.

My impression is that they largely seem to focus on Big Money HPC contracts and other very large deals where bespoke engineering on the software stack is worth their time and can be paid off. So if you're paying them a shitload of money in a massive supercomputing-style deal, they'll get you software for their otherwise-unobtainium accelerators. But normal "day to day" stuff is literally not on their radar. At best it's an afterthought.

I think the timing of when they entered into the field is also relevant. When Nvidia first came on the scene swinging, they really needed to see what would stick; so you have to try fishing in every available pond. You don't know what will necessarily pan out or catch on. So you need the tech to be available to all your consumers at every level, consumer, enterprise, HPC, everything. And it turns out, they found that there are customers at every single one of those levels, and providing for all of them provides both a good on-ramp, and a path to bigger things.

In contrast AMD is so late to the game, and there's so much money sloshing around in the field right now, and the trajectories of what big buyers want is so much more clear -- that they can just afford to mostly ignore the lower end. They know where the money is. The field and buyers and use cases are just much more well understood from the business "make money in the easiest way" POV.

As for Intel, they seem to actually be dedicating tons of money to the software stack for oneAPI, to work everywhere, which is not surprising IMO. They have the money to dump into this (the most money intensive) task and know it's critical for their products to reach anything other than pork-belly HPC contracts.

It's still peanuts compared to what semiconductor research&engineering costs. They can afford it to make the most out of their market share. AMD must be rational and prioritize to repay investors for their trust. Time will tell whether that strategy works out.

I feel there's a large disconnect at AMD between the CPU division and the RTG (Radeon Technologies Group) which feels more insular than the rest of the company. I feel Lisa Su had her hand deeper in propping up the wailing CPU division and kind of forgot about RTG that's been loosing market share to Nvidia for the last 10 years not just in gaming but in ML. Maybe she should take a break from Ryzen and look into leading the RTG.

Also AMD seems to underestimate how many people may just be curious to use compute functionality with their consumer based card. I was ultra excited to get a rdna2 card to make a dedicated linux system with a friendly gpu. Gaming works, but davinci resolve doesn’t, can’t use AMF for screen recording or streaming. I have a 3060ti back in a box abd parts for a “work computer,” and I thought I would work a bit with my system with the 6700xt but can only really game, screen record, and edit video with kaden live using my intel cpu.

I watched a video over the weekend about how a new amd apu using rdna3 will make entry level/cheap graphics cards useless, but thats not really going to be the case if you want to dip your feet into utilizing an entry level card for workloads. Gamers turn their nose up at dinky gtx 1650s but at least you can use it to edit video. Theres no wonder intel doesnt see AMD as real competition in the space.

> Nothing actually works on non-NVidia hardware. "Actually works" is different from "theoretically works." A lot of what I use theoretically supports ROCm, but in practice, when you try to use it, you have enough bugs big and small, and things crash or don't work right.

I was surprised when I started playing around with Stable Diffusion a couple of weeks ago and it just kind of worked with ROCm. That being said, I did run into a couple of very annoying issues... almost running out of VRAM seems to trigger GPU timeouts but as far as I can tell the GPU itself is fine and the memory allocation crawled to a halt. This also seems to be much more likely when running at half precision, which is a shame given that the relatively large 16GB VRAM effectively gets cut in half when running at full precision...

But this is already a big improvement over "ROCm doesn't support RDNA2" [0] which was still the case ~1.5 years ago. I just really hope that AMD keeps up the effort and fixes all the small and big issues that are still around...

[0]: https://github.com/RadeonOpenCompute/ROCm/issues/1180

> I was surprised when I started playing around with Stable Diffusion a couple of weeks ago and it just kind of worked with ROCm.

Stable Diffusion is literally the single highest-profile application of machine learning. There has been extensive effort to make it work in all sorts of places. If it were the year 2000, someone would port it to a Beowulf cluster...

I'm more concerned about SpaCy, CuPy, the hundreds (thousands) of models on Hugging Face, etc. I do use a lot of that stuff. Buying a GPU or accelerator which only runs the high-profile stuff is a bad buy.

The other models on Hugging Face I tried also worked fine... I'm not saying that everything is running perfectly or even that buying an AMD card for compute is a good idea at the moment, but there has been quite some progress even compared to 2-3 years ago so I'm cautiously optimistic...

I still remember trying to train a model with TensorFlow on AMD maybe 2 years ago. The only option was OpenCL and it was disappointingly slow (and barely any load on the GPU) and in the end I switched back to my old 1060 which worked much better.

ROCm also has significant startup latency, and for Stable Diffusion additional latency compiling the new operations whenever you switch resolutions.

I found that running out of memory would lock up my X session, spam kernel logs about GPU ioctls timing out, and require a reboot. That's much less friendly than CUDA reporting an error to the requesting application and carrying on!

Another thing that could happen is a fundamental algorithmic advance to replace backprop. The brain doesn't use it, and backprop doesn't have answers for how to do long term credit assignment and continuous learning. It works poorly with RL as well.

Better algorithms would require writing new ML libraries which would weaken the stranglehold GPUs have on ML. New niches opening would inevitably be bad for Nvidia, but good for upstarts.

Geoffrey Hinton recently came up with a new "forward forward" algorithm that seems to fit the bill:

https://bdtechtalks.com/2022/12/19/forward-forward-algorithm...

I recently migrated a lot of my CUDA code to SYCL using Intel's compatibility tool [1]. I'm trying to get my applications to work irregardless of hardware, so it simply needed to be done despite the pain.

It actually wasn't as hard as I was expecting. Don't get me wrong, it will definitely depend on the size of your code base as to how effective of a tool it is. However, I managed to port over most of mine in the course of the month, and it's showing dividends now I'm making back performance gains using FPGA hardware acceleration. I'm now vendor neutral, and can run my code on team Red, Blue, or Green. I'd argue that heterogeneous computing is a good enough reason to make the switch.

[1]https://www.intel.com/content/www/us/en/develop/documentatio... [2]https://www.khronos.org/sycl/

I'm only an interested observer of the space, but a bunch of work is being done getting at least some of this stuff to run on Apple Silicon in recent months.

I've been running Stable Diffusion on my M1 MBP since...I dunno, November or something? and it's working great.

Slow as heck tho compared to on a dedicated card. I was running it on my M1 pro till I bought some old miners to resell and tried out a 3070. That thing gets me a 512 x 512 DDIM 20 steps photorealistic image with protogen 5.3 in 2.85 seconds. Its insane to watch it churn out images like that. My on my MPB I had to downgrade to the older SD model that was trained only one 256 x 256 to get decent feedback on my prompts.

Really? Mine usually only takes about 30s per image for 50-step 512x512 images (which is perfectly reasonable for my purposes)...I wonder if it has to do with the RAM (I've got an M1 Max with 64GB)? Or maybe I'm just not doing it enough to care about the speed so much.

Compared to the 2.85 seconds on my dedicated card? Yes 30 seconds is slow as heck. Youre talking about 10 times longer and waiting this time to adjust your prompt.

I dont find that acceptable for a workflow where I need to tweak prompts.

> Slow as heck tho compared to on a dedicated card.

It's important to mention that there is a huge difference in performance between a new MacBook with 8 GB or 16 GB of RAM when it comes to this memory-intensive ML tasks.

I got the 16gb version, though its too slow for a real workflow even on the heaviest model. Its literally a magnitude of difference going from 30 seconds to less than 3.

I dont think the m1 pro comes with 8gb either, so kinda unwarrented assumption. I love the kneejerk reaction on me saying anything bad about apple though.

Critically, there's also cost. The formula is $W Nvidia for N-flops vs $X ROCm for N-flops + $Y engineering cost of recoding.

If $W > $X + $Y, then people will go with AMD.

That's roughly right only if the people paying $W vs $X and $Y are the same people. The people paying $X and the people paying $Y are often not the same people across large swaths of the GPU-buying ecosystem. ($W and $X also change over time and with volume of units shipped and volume shipped varies with how useful and widely applicable the cards are.)

I'd love to get an AMD [6650XT-ish] for my teen's gaming PC. But they also do some programming with me and I slightly worry about "oh, your PC can't do CUDA, therefore can't do XYZ that's interesting to you". I probably will still just go AMD and then have us use rented GPUs if we want to do any CUDA demos together.

While the hardware and software budgets are not the same, some companies do have tricks to make the software team care about hardware at least in theory.

I have never ordered any server hardware, but there's a dashboard where I can see some abstracted infrastructure CAPEX and OPEX numbers for my team, org, division, etc.

These numbers get reported up the food chain and initiatives to reduce costs are rewarded if the savings are significant enough.

Initiatives to support ARM servers or AMD's CUDA equivalents can take a long time and be expensive to achieve full compatibility, but experiments in areas where the effort is not terribly high can demonstrate enough value to fund teams to do so.

I suspect that's most difficult in mid sized organizations where the hardware costs are high enough to get some savings but not high enough to fund a team to work on it, especially once the opportunity cost of new features is taken into account.

Universities, mid sized companies, big companies without a software focus, etc. spring to mind as the ones with the biggest disconnect between hardware costs and ability to fund engineers to cut them down.

For the long tail of ML uses, $Y dominates.

That's actually most of the market. AMD will do fine for 3d gaming, stable diffusion, and a few similar uses. And price/performance wins for giant data centers doing something specialized (but if NVidia is about to lose a major data center deal because $W > $X + $Y, they can just cut a deal on $W).

A great report, which concludes with “everything we described here only supports CUDA backend for now, but it will change at some indeterminate point in the future”.

And there’s a reason for that. OpenCL and particularly ROCm are much lower level and much less immediately usable compared to CUDA. AMD now has HIP, but HIP is “something that is almost CUDA-compatible, with no real improvements”, instead of actually doing something new and better. CUDA is an old API, and certainly has its rough edges, but I wonder why nobody has managed to do something genuinely _better_ in all these years.

ROCm is so low-level and pedantic it is nearly unusable. A simple implementation of FFT takes about 150 lines of C++/CUDA and about 2000 lines in ROCm — so no, no competition at all.

One of the best CUDA decisions was to go polyglot, and in that context, first class support for Fortran.

The competition is still like "Fortran what??!" every time I happen to watch a Khronos or AMD talk on compute.

I remember people saying the same of the Mantle/Vulkan API when compared to DirectX. Now it powers gaming on Linux and the Steam Deck, and now even Intel uses DXVK in their new Arc drivers.

ROCm is a framework, and its existence likely has a bigger impact on the quality of CUDA and the good behavior of Jensen than we realize.

While being mostly ignored everywhere else.

Enjoy your 2% market share.

I know essentially nothing about any of these tools, but I get the impression that the report is arguing that the API for ML is really up at the Python layer, and ML programmers don't interact directly with the stuff you're talking about.

If the new PyTorch funnels everything through abstractions that don't require you to implement as many primitives to port it onto a new backend, then it matters less than before if ROCm (or whatever...) is harder to use, because the person doing the port don't have to do as much with it, and the actual application programmer never sees it at all.

Yes, but the assumption that python abstracts from this, and that ML programmers don't interact with cuda is wrong.

Almost the entirety of the post is not related to the title. The summary is of this is given in last:

> OpenAI Triton only officially supports Nvidia GPUs today, but that is changing in the near future. Multiple other hardware vendors will be supported in the future, and this open-source project is gaining incredible steam.

It's not even the first tool for allowing ML workloads in non NVidia cards. ROCm already exists. TF already supports M1/M2 GPU already. The entire point is that no cards come close to Nvidia on raw GPGPU performance, and this is not going to change in near future.

Have you actually tried to write something with ROCm compared to CUDA? For me, ROCm is very low-level, boilerplate-ridden and nearly unusable.

HIP is very similar to cuda. There's a perl string substitution thing that will try to convert one to the other.

There is lower level code below cuda and below hip. In cuda, it's closed source and hidden. In rocm, you can follow the control flow all the way into the amdkfd kernel module.

I like having the option to program at a lower level than cuda. When I need to write in hip, I like having all the source from the stack trace.

Except the small detail that CUDA is polyglot and quite a few runtimes target PTX.

Those Perl scripts won't do nothing for them.

In which sense is cuda polyglot? It's a dialect of C++. Maybe you mean cuda the toolchain (driver libs, ptxas etc) as opposed to cuda the language. Shame they have the same name.

Anything targeting PTX via LLVM should be able to target GCN by changing flags, though that may not be totally seamless.

Things writing ptx assembly directly would be more awkward to retarget, there's no direct equivalent there.

In the sense that it officially supports C, C++ and Fortran on NVidia/PGI toolchain, plus anything else that targets PTX and there are plenty of those, like .NET, Java, Haskell, Julia.

It is only a dialect of C++ for those that don't bother to read about what CUDA actually is.

https://developer.nvidia.com/language-solutions

https://developer.nvidia.com/hpc-sdk

I don't use CUDA or ROCm. I use tensorflow/pytorch which supports both of them. But the performance difference between them is huge, as AMD cards have much lower GPGPU/Tensor flops compared to top NVidia cards.

Any good benchmarks on large transformers?

In my opinion the only thing that would break the NVIDIA monopoly is if some other vendor (maybe Intel) sells a graphics card that breaks NVIDIA's consumer/datacenter divide (e.g. offers A100-size memory for consumers to purchase, ideally less than $1k), and starts working on 100% PyTorch compatibility with their software extension. No custom kernel patches, no Perl substitution, I want to (a) install a driver, (b) pip install a model, and (c) run it.

The vast vast majority of ML researchers do not want to spend time fiddling with compiler flags. Give them something they can't do with NVIDIA, make the process not awful, and there's money to be found. At the very least, "winning" at AI will increase your revenue multiple...

Nvidia already does sort of break that divide, don’t they? By offering cuda on consumer cards, they offer a gateway to curious users, and something to get by with at home for professional users. I think intel was late to the game with Arc because they insisted on making a card that could compete with 80-90% of nvidias features on their first mass market try, and had to build much of that infrastructure from scratch.

Excellent. Die, nVidia.

I will never buy another nVidia card. They have made the firmware only modifiable to themselves. This forbids you from slowing down the card, for instance to run at a lower temperature (and prolong its life).

"Little hope of reclocking becoming available for GM20x and newer GPUs as firmware now needs to be signed by NVIDIA to have the necessary access."

https://nouveau.freedesktop.org/

> They have made the firmware only modifiable to themselves

The same is true for current generation of AMD cards :/ You can no longer edit voltage tables

I didn't think that was the case. Do you have a reference?

I don't believe the firmware build tools are public. If you want to have a go with a hex editor, I'm not aware of any encryption or code signing going on. Interested to hear if I've missed it though.

https://www.hardwaretimes.com/amds-radeon-rx-7900-series-gpu...

Thanks for the link. I think that's a different thing - there's something involving changing settings in the windows registry via some vendor tool where an option as been removed from the GUI. There's a reference to that as an alternative to "flashing modded BIOSes" but no claim that the BIOS can't be modified. Do GPU's have a BIOS? I thought that was a (now legacy) x86 thing.

Side note, the cookie popup box on that website was ridiculous and lacking the 'reject all' option, but the content is readable by disabling javascript.

Ah, well, only second-hand for me then. Thanks for letting me know.

Hopefully my laptop will outlast these tyrannical practices. If it doesn't, I'll just skip the dedicated GPU entirely.

I dislike the nvidia experience on linux as well, but you can undervolt nvidia cards on windows (and yes that makes me cringe).

Kinda funny we went from the wintel era to the winvidia era

The bottleneck is memory bus bandwidth as current Neural Net scaling is parameter count over compute so most A100s only operate at 60% of capacity.

Operator fusion ( function composition ) has helped but devs time is impacted by the ever growing list of fused operations.

This is a great overview of what Pytorch 2.0, TorchInduction, and OpenAI's Triton potentially bring to the table.

Namely compilation to LLVM bypassing CUDA, fusing operators and compiling graphs at a lower level allowing other arcitectures, kept out by CUDA's NVIDIA exclusivity, to potentially offer competeting compilers - other compilation toolchains are very early days though.

Backward comptibility is vital and the article details that these innovations are succesfully tested against existing Neural Net code.

NVIDIA still rules the roost in science and industry compute. In this rapidly growing market competition is healthy so this is a good thing for everyone - especially those differentiators like ARM competing on Watts per FLOP.

It is quite weird to talk about all the frameworks which are built on top of CUDA eventually, but not talking about Rocm or OpenCL.

OpenAI's Triton compiles down to CUDA atm (if I read their github right), and only supports Nvidia GPUs.

PyTorch 2.0's installation page only mentions CPU and CUDA targets, therefor it's effectively all Nvidia GPUs.

While all the frameworks and abstractions could offer other back-ends in theory the story of anything ML related on the other big name in the industry, AMD, is still poor.

If anybody looses business because of bad decisions it is AMD, not Nvidia, who lead the whole industry. I am not convinced that anything will change in the near future.

From https://news.ycombinator.com/item?id=32904285 re: AMD ROcm, HIPIFY, :

> AMD ROcm supports Pytorch, TensorFlow, MlOpen, rocBLAS on NVIDIA and AMD GPUs: https://rocmdocs.amd.com/en/latest/Deep_learning/Deep-learni... [...]

> ROCm-Developer-Tools/HIPIFY https://github.com/ROCm-Developer-Tools/HIPIFY :

>> hipify-clang is a clang-based tool for translating CUDA sources into HIP sources. It translates CUDA source into an abstract syntax tree, which is traversed by transformation matchers. After applying all the matchers, the output HIP source is produced. [...]

From https://github.com/RadeonOpenCompute/clang-ocl :

> RadeonOpenCompute/OpenCL compilation with clang compiler

A better overview from the docs: "Machine Learning and High Performance Computing Software Stack for AMD GPU" https://rocmdocs.amd.com/en/latest/Installation_Guide/Softwa...

AMD's position on machine learning has been fascinating to watch. While the company has gone through some cash strapped years, investing even some marginal effort into the ML space would have paid off handsomely in data center sales. Instead they let nvidia run away with it, pushing out a half-baked ROCm, which bizarrely supports such a tiny subset of cards it is farcical: With nvidia devs with a standard consumer GPU are building solutions and testing models, and when it's time to push big staying with nvidia is just the easy choice, while AMD apparently tries to upsell for even marginal support.

Apple has done more to see Apple Silicon supported in the ML space in almost no time than AMD did over the past decade. Apple obviously has more resources, but they don't have a 1000 person team working on this, I suspect, but instead it's a very commando, efficient operation. They got a small number of dedicated resources committed to bringing support and enabling the community, and it happened.

Shameless plug, I wrote a similar post a few months ago focusing more on the APIs side of things:

https://www.the-investing-desk.com/nvidia-will-lose-its-grip...

Could we say that nVidia is slowing down innovation for their own profit?

On the SW side, yes.

"skipping Nvidia’s closed-source CUDA libraries, such as cuBLAS, in favor of open-source libraries, such as cutlass"

Eww, so it'll be 90% slower?

cuBLAS is insanely optimized. It's the reason why CUDA dominates the field even though one could in theory use a ROCm build of TensorFlow or PyTorch. But performance is abysmal without cuBLAS. So the entire article appears to hinge on a huge assumption: What if Free Open Source was equivalent to NVIDIA's top-tier priorietary library? (It's not)

Cutlass is from NVidia and AFAIK gets pretty close to cuBLAS performance. I think the use case for cutlass is when you only need a few kernels and don’t want to pull in a huge cuBLAS dependency and are ok paying a small perf penalty for that. Then there are times when you need custom kernels that not available in cuBLAS and for that cutlass is about as fast as it gets.

cutlass is faster actually, in most cases where cutlass supports same stuff.

I presumed they’d fold cutlass improvements into cuBASE so cuBASE would be at least as fast. Maybe some compile time optimizations?

https://archive.ph/upoNg

You do know that this is a just a capture of the free part of the post? (ie most of it)

Readers should at least visit the original site if they want to read the free part.

Thanks! Ya, the archive link is useless, it only captures the free part, and I think I am very generous with what I keep in the free section on the ad-free website.

Im curious, what stops AMD writing an api/abi compatibility layer for cuda?

The fate of ZLUDA points to the existence of some kind of legal obstacle: https://thenewstack.io/zluda-a-cuda-for-intel-gpus-needs-a-n...

https://rocmdocs.amd.com/en/latest/Programming_Guides/HIP-po...

Lawsuit risk from nvidia seems likely

My impression: arrogance

The whole ROCm stuff is built around the idea that you push out a half-working, buggy, and slow mess as Open Source and then the community will take over and maintain it into a usable product. But that hasn't happened so far. Instead of building their own FOSS libraries, people are happy to use NVIDIA's proprietary libraries.

Deleted comment

> Unfortunately, whether you use CUDA or not, NVIDIA hardware is still the only realistic performant option.

> Maybe the move away from CUDA dependence for these frameworks will make other manufacturers step up competition for accelerating ML workloads. But they should have been doing that anyway...

That is entirely down to SW, not the HW. For example consider these two 24GB cards:

Nvidia 4090: 82.58 TFLOPS/$1,599 = 51 GFLOPS/$

AMD 7900 XTX: 61.42 TFLOPS/$999 = 61 GFLOPs/$

> It's easier to debug Python, largely since it's interpreted.

Uhh, it's easier to debug because it's not C++. Rust is fairly easy to debug and profile, despite being compiled

C++ is easier to debug than Rust. Rust debuggers are very limited with expression evaluation. Specifically, trait calls are still impossible AFAIK.

While I don't have an installation around me now, I am sure I could do that in CodeLLDB/VSCode.

Reply on news.ycombinator.com