Comment by tialaramex | Hacker News Reader

Comment by tialaramex | Hacker News Reader

Parent Full thread

Huh, that Wiki link taught me something I didn't know - I hadn't realised a different FP representation could have Denormals (a word I knew) which weren't Subnormal (a new word to me, more precisely defining what's different about these values).

The Subnormals (presumably any Denormals regardless?) tend to be annoyingly slow, because the FP hardware usually special cases them to some slow but correct silicon not a fast path for the usual case. So, in some code it can make sense to "zap" subnormals, converting them to zero since they're almost zero anyway. Obviously in other code this is a terrible idea, so your domain knowledge comes into play when authoring such code. For example if we're using -1 .. +1 as the 0dB for our Karaoke software, a subnormal is clearly inaudible noise, but it might make our sophisticated reverb simulator (so you can pretend you're singing in a big auditorium) a thousand times slower than normal, so, zap those subnormals before the simulator and we hear the same results but don't get punished with miserable performance.

Denormal and subnormal are synonyms as far as I know and can find. That first sentence is awkwardly phrased: "subnormal numbers are the subset of denormalized numbers (sometimes called denormals)" would be better as "subnormal numbers (sometimes called denormals) are the subset of denormalized numbers". It does imply there are other "denormalized" numbers though; not sure what they are.

Terminology is confusing!

A floating-point number is generally defined as (-1)^sign * (mantissa) * base^exponent, where mantissa is a sequence of digits in the base.

A normalized floating-point number has mantissa be a fixed number of digits for a given floating-point type.

A denormalized floating-point number is one with a different number of digits, where the exponent is the minimum possible value.

An unnormalized floating-point number is one with a different number of digits, but the exponent is not the minimum possible value. (This doesn't occur in the standard IEEE 754 types, but the PPC and the x86 long double floating-point types have these numbers).

There's also noncanonical floating-point numbers, which are different representations for the same value. These occur, for example, with IEEE 754 decimal FP, as well as many pre-IEEE 754 types (where instead of subnormals, you tend to get noncanonical representations of 0). There's also another kind of scenario where you have multiple values that are all still canonical--for example, 0.0 and 0.00 are different canonical decimal FP values in IEEE 754 (they have different exponents, and I'm not going to attempt to remember what they are off the top of my head), and of course -0 and +0 are distinct values of 0 that work differently.

You didn't cover what exactly "subnormal" means

In all practical terms, subnormal is the same as denormal (a non-normalized number with the smallest possible exponent).

I think this is a situation where there's confusion over precision of terminology, so things have shifted to a new term which is consistently used. Subnormal is now the new preferred terminology for denormal, I think out of confusion as to whether or not unnormal numbers were also denormal (I have never heard the term so used, but I'm also not a great connoisseur of non-IEEE 754 floating-point).

There's a similar attempt to shift use of the term "mantissa" to "significand", since a mantissa implies that the number is in the range [0, 1), whereas the usual implementation in a floating point type is to use the range [1, base).

this is wrong(!)

> A normalized floating-point number has mantissa be a fixed number of digits for a given floating-point type.

well, no. in fact...

> A normalized [binary] floating-point number has mantissa which is bit shifted so that the first bit to the right of the binary point is '1'.

(not a great look in an expert explainer comment)

Erm, no, not in 754 floating point.

The significand of a normalized number starts with an implied 1 to the left of the decimal point, which isn’t explicitly represented. In other words, the significand represents the range [1, 2) for normalized numbers.

In fact, that’s reason we need subnormal numbers: they are numbers for which the implied first bit is not 1, so the significand represents the range [0, 1).

Subnormal numbers therefore do not have a fixed number of digits (precision). As subnormals approach zero, the precision gets smaller.

fair point - I did reference this from a general article and it did not specify P754 - I guess this has an implied 1 and can save one bit of data path … but the definition of normalized is that the mantissa is shifted to be aligned to the binary point and not that is have a fixed number of digits since it will often be packed with trailing zeros

> but the definition of normalized is that the mantissa is shifted to be aligned to the binary point

And that’s the key point. A subnormal number does not have the significand/mantissa aligned to start with an (implied) 1. It has leading zeros. Thus the number of digits of a subnormal number is not fixed.

1.0000x10^0 has the same precision as 1.0000x10^-99.

1.0000x10^-99 does not have the same precision as 0.0010x10^-99.

That is why the original poster wrote the subnormals have a not-fixed number of digits.

But if you’re going to go around writing snarky “gotcha” comments, at least try to get your facts straight.

For IEEE 754-2019 basic formats, subnormal and denormal are synonyms.

There are formats (like x87 80-bit as implemented in pre-387 hardware¹) that are not 754 basic formats which admit non-normal encodings that are not subnormal (i.e. not smaller in magnitude than the smallest normal number). Some people used to call these "denormal" as well (even though the x87 docs call them "unnormal"), which caused confusion. "Subnormal" is unambiguous.

In decimal IEEE 754 formats there are also non-canonical encodings, but these are neither subnormal nor denormal (decimal encodings are not, in general, normalized the same way binary floating-point is).

-----

¹ on 387 and later these are invalid encodings, and behave like a signaling NaN.

> So, in some code it can make sense to "zap" subnormals, converting them to zero since they're almost zero anyway.

Flush to zero (FTZ) is the usual term for this.

> FP hardware usually special cases them to some slow but correct silicon

This is basically only true on Intel chips. Subnormals are not that slow on modern ARM or AMD hardware.

This is only mostly true unfortunately. If I recall my tests on M1 M1 has a performance penalty for subnormals, but more like a 10x than the 200x that it causes on Intel.

Reply on news.ycombinator.com