Hacker News Reader

The catch is it's only for TF32 computations (Nvidia proprietary 19 bit floating point number format)

I missed that, to me that makes the ‘32’ in the name misleading.

TF32 = FP32 range + FP16 precision

Why not call it TF19 then.

Because your existing FP32 models should run fine when converted to TF32, so TF32 is equivalent to FP32 as far as DL practitioners are concerned.

There is a lot of redundancy in DL that forgives all manner of sins; think it’s sneaky.

Because it's 32-bits wide in memory.

The effective mantissa is like FP16 but it's padded out to be the same size as FP32.

In other words, there's 1 sign bit, 8 exponent bits, 10 mantissa bits that are USED, and 13 mantissa bits that are IGNORED.

1 + 8 + 10 + 13 = 32

The 13 ignored mantissa bits are part of the memory image: they pad the number out to 32-bit alignment.

But the user never sees that memory right? Doesn't it go in FP32 and come out FP32? I still think it's deceptive marketing.

The user does see 32-bits and all bits are used because all the additions (and other operations besides the multiply in matrix ops) are in FP32. So the bottom bits are populated with useful information.

And with 50% sparsity

Reply on news.ycombinator.com