Ask HN: Does Nvidia have any credible competition in the AI/DL space? | Hacker News Reader

Ask HN: Does Nvidia have any credible competition in the AI/DL space?

22 pointsmmaunder15 comments

Some are here - some of these are things people are using today, some are available but don't have much user adoption, some are technically available but very hard to purchase or rent/use, and some aren't yet available:

* Software: OpenAI's Triton (you might've noticed it mentioned in some of "TheBloke" model releases and as an option in the oobabooga text-generation-webui), Modular's Mojo (on top of MLIR), OctoML (from the creators of TVM), geohot's tiny corp, CUDA porting efforts, PyTorch as a way of reducing reliance on CUDA

* Hardware: TPUs, Amazon Inferentia, Cloud companies working on chips (Microsoft Project Athena, AWS Tranium, TPU v5), chip startups (Cerebras, Tenstorrent), AMD's MI300A and MI300X, Tesla Dojo and D1, Meta's MTIA, Habana Gaudi, LLM ASICs

I'm in the process of writing about this (I'll probably post in the hiring freelancer thread tomorrow, might like to find a freelancer to help me research and write it).

The A/H100 with infiniband are still the most common request for startups doing LLM training though, I did some research and a writeup on that.

If I'm missing or miscategorizing any above, please let me know.

Side note: The current angle I'm thinking about for the post would be to actually use them all. Take Llama 2, and see which software and hardware approaches we can get inference working on (would leave training to a follow-up post), write about how much of a hassle it is (to get access/to purchase/to rent, and to get running), and what the inference speed is like. That might be too ambitious though, I could see it taking a while. No points for companies that talk a big game but don't have a product that can actually be purchased/used, I think - they'd be relegated to a "things to watch for in future" section.

Currently there’s lockin due to most things calling upon cuda on nvidia gpus. There are alternatives to cuda but they’re up and coming: triton, tvm, openvino, improved rocm, etc.

I think AMD is at the very least, credible.

https://wccftech.com/amd-instinct-mi250-boosted-ai-performan...

I think they are better than people give them credit for. They seem to have a stigma against them.

Nobody wants to make a large deployment of AMD hardware because there are of course risks involved and nobody wants to be the one to blame if it doesn't work out. I can't blame them, but that doesn't mean it wont potentially still work out.

All it takes is for a big successful deployment I think.

> All it takes is for a big successful deployment I think.

Yes - I think one company doing a big deployment of AMD GPUs for training and inference would be extremely confidence boosting for other companies.

I wonder why AMD doesn't do it themselves.

If investors ask me whether to build AI supercomputer using AMD cards (rather than Nvidia), today I would say no.

why?

My employer is getting closer: https://www.hpcwire.com/2023/06/28/mlperf-training-3-0-showc...

From your link:

“Only two chips took on the LLM challenge – Nvidia’s H100 GPU and Intel/Habana’s Gaudi2 deep learning processor – each showcasing different strengths. Not surprisingly, the newer H100 was the clear performance winner.”

I wouldn’t say Intel is getting close, and to actually beat Nvidia any competitor needs to be much better (not just “getting close”).

To beat Nvidia, you have to solve software. The rest is negotiable - weaker hardware can still be attractive with more competitive pricing or efficiency.

Nvidia is definitely on top of their game right now, but only because they've gone all-in betting on ecosystem fragmentation (and keep getting proven right). Sufficient software investment could unseat them handily.

Nvidia is definitely on top of their game right now

Nvidia has been on top of their game since at least 2012, when I shopped for ML hardware for the first time. AMD had plenty of time to “solve software” - what happened?

> what happened?

Key stakeholders in OpenCL started pulling out. The upcoming Khronos spec didn't have the entire industry on board, and both Microsoft and Apple pushed vendor-controlled GPU runtimes on consumer hardware. AMD didn't have the same support they did pre-2010, so a lot of their efforts fell apart.

Again, Nvidia's key is that they thrive in this chaos. They want people to make half-assed Metal Performance Shaders and DirectML programs, because it makes their ecosystem seem so strong and intersectional by comparison. If Intel invests hard enough in the right software, that's half of the problem solved.

For inference, yes. For training, no.

Google trains on TPUs.

Cerebrus wafer

Google TPU

Not that these are for the average buyer

Reply on news.ycombinator.com