Hacker News Reader

Hacker News

Comment by Const-me | Hacker News Reader

Parent Full thread

Const-me

> Can you link to benchmarks or other analysis supporting this claim?

Current generation nVidia: https://en.wikipedia.org/wiki/GeForce_40_series#Desktop

Current generation AMD: https://en.wikipedia.org/wiki/Template:AMD_Radeon_RX_7000

The key performance characteristics are processing power TFlops, and memory bandwidth GB/s.

nVidia 4080 which costs $1200: 43 TFlops FP16, 43 TFlops FP32, 0.672 TFlops FP64, 717 GB/s memory.

AMD 7900 XTX which costs $1000: 93 TFlops FP16, 47 TFlops FP32, 1.5 TFlops FP64, 960 GB/s memory.

Note that for applications which bottleneck on FP16 compute (many ML workloads) or FP64 compute (many traditional HPC workloads: numerical solvers, fluid dynamics, etc), the 7900 XTX even outperforms the 4090 which costs $1600.

View on HN

In ML workloads, usually the FP16 operations are matrix operations. On RDNA3, these execute at the same rate as normal shader/vector operations, but on Nvidia RTX cards there are Tensor cores which accelerate them. The Ada whitepaper lists 48.7 shader TFlops (not 43 because boost vs base clock), and 195 TFlops for FP16 Tensor with FP16 Accumulate. That's 4 times faster than regular, and almost double what the XTX lists!

Ampere and newer also have native sparsity support which means that you can skip over the zeroes 'for free', which Nvidia uses to market double the TFlops, which is kind of misleading imo. But the 195 TFlops are even before sparsity is included!

I'm not sure if the 93 TFlops (120 with boost clocks) on AMD are with FP16 or FP32 accumulation, as with FP32 accumulation the 4080 slows down significantly and gets much closer with 97.5 TFlops.

Intel Xe-HPG (used in the Arc A cards) also offers very aggressive matrix acceleration via XMX, with 137.6 FP16 TFlops at base clock, vs. 17.2 FP32 TFlops.

You’re comparing general-purpose computations to a proprietary feature with limited applicability. For example, in my Mistral implementation the most expensive compute shader is handling matrices compressed with a custom codec unknown to tensor cores.

This is not an apples-to-apples comparison. We use FLOPS numbers to compare processors of different architectures because they’re the same FLOPS everywhere.

>This is not an apples-to-apples comparison.

Neither is FLOPS. RDNA3 FLOPS numbers are greatly inflated, because they are only applicable to fairly specific VOPD and WMMA instructions, the former being for two MADs at the same time, and the latter being applicable in the same cases as tensor cores.

Besides, it should be possible to use the tensor cores combined with the codec: you can use vector hardware to decode matrix values to fp16/fp32, then yeet those into the tensor cores. Although most of the time will probably be spent on the decoding part, assuming you're doing matrix-vector multiplication and not matrix-matrix (which might be different with MoE models like Mixtral 8x7B?)

RDNA3 FLOPS numbers are real. Shader compiler happily uses VOPD for quite a few things including FP32 FMA.

An independent researcher measured 62.9 FP32 TFlops (boost) in Vulkan on 7900XTX, see “Full GPU Throughput – Vulkan” section on that article: https://chipsandcheese.com/2023/01/07/microbenchmarking-amds...

That is interesting, even though I would need to look at non-consumer-GFX cards. What about actual benchmarks, though? Passmark, for example:

https://www.videocardbenchmark.net/directCompute.html

has:

GeForce RTX 4090 28,916

RTX 6000 Ada 23,296

GeForce RTX 4080 22,016

Radeon RX 7900 XTX 18,767

and Geekbench 6:

https://browser.geekbench.com/opencl-benchmarks

has:

GeForce RTX 4090 321928

L40 292357 <- L40S is out and may be better

RTX 6000 Ada 274348

H100 267514

NVIDIA GeForce RTX 239863

Radeon RX 7900 XTX 199412

Neither of those benchmarks is relevant to ML which is what GP is talking about

So what is relevant? As other poster said, nVidia tends to outpace AMD cards in pretty much every use case in real life.

The only important numbers are processing power TFlops, and memory bandwidth GB/s.

If your compute shader doesn’t approach the theoretical limit of either computations or memory, it doesn’t mean there’s anything wrong with the GPU. Here’s incomplete list of possible reasons.

● Insufficient parallelism of the problem. Some problems are inherently sequential.

● Poor HLSL programming skills. For example, a compute shaders with 32 threads/group wastes 50% of compute units of most AMD GPUs, the correct number for AMD is 64 threads/group, or a multiple of 64. BTW, nVidia and Intel are fine with 64 threads/group, they run 1 thread group as 2 wavefronts which does not waste any resources.

● The problem being too small to compensate for the overhead. For example, CPUs multiply two 4x4 matrices in a small fraction of a time it takes to dispatch a compute shader for that. You gonna need much larger matrices for GPGPU to win.

Reply on news.ycombinator.com