Comment by adrian_b | Hacker News Reader

Comment by adrian_b | Hacker News Reader

Parent Full thread

Denormals are not needed if you are willing to handle underflow exceptions.

Before Intel 8087 and the IEEE 754 standard, any decent floating-point unit generated overflow exceptions and underflow exceptions, which had to be handled by the programmer, unless the default behavior of crashing the program was acceptable.

Intel 8087 and the standard based on it have offered to the lazy programmers the option to not handle the exceptions, in which case overflow exceptions generate infinities and the underflow exceptions generate denormals.

When the exceptions are not handled, it is supposed that the programmer will check the final results of a long computation, and if infinities and denormals are not desired, but they exist nonetheless in the results, the programmer will investigate the reason and then the bug will be fixed.

So anyone is free to ensure that no denormals will ever appear in an application , by enabling the underflow exception. If it is desired that the program must not crash, then the program must be written carefully, so that underflows are impossible.

There is no correct way of eliminating denormals, except throwing exceptions on underflows.

The flush denormals to zero on output and interpret denormals as zero on input behaviors are not permissible in any program that must produce correct results. Anyone who uses -ffast-math or similar options for compiling a program that is not intended for graphics or ML/AI, where errors supposedly do not matter, makes an unforgivable mistake.

Unfortunately, "-ffast-math" enables a very large number of compilation options. A part of them are safe and they can cause a great increase in performance, like using fused multiply-add instructions. Others not only are dangerous, like flushing denormals to zero, but they also provide negligible performance increase on many processors.

Therefore, instead of aggregate options like "-ffast-math" one must enable only some of the component options, for maximum performance, without affecting result accuracy. For example, in gcc and clang one must use "-ffp-contract=fast", for enabling FMA instructions.

Sorry I did not understand the point about underflows. Underflow happens with and without denormals - denormals just allow to go a little further before the number turns into a zero.

Also, there are cases when you do not want to crash on overflow - for example, live audio processing, you do not want sound to stop only because there ws an overflow in one audio sample.

When underflows happen there are 2 possible actions on any standard processor, which are selected by a bit in a configuration register.

One option is to throw an exception. In that case a program will never generate any denormals. Denormals can appear in such a program only when they are present in input data generated by another program, which has been executed with a different exception configuration.

The second option is to generate denormals. In such a configuration every underflow generates a denormal and such denormals are propagated through later arithmetic operations, unless they are added to or subtracted from a much bigger number, when they disappear.

What you say about audio is the very reason why denormals have been introduced in the standard. You are allowed to choose the second option, i.e. to mask the underflow exception, so there will be no underflow exceptions and the underflows will generate denormals, which will introduce minimal errors that are acceptable in most applications.

Unlike with the errors introduced by denormals, which are typically negligible, the non-standard third option, to flush denormals to zero, can easily produce huge errors that are unacceptable for most applications. Thus this non-standard option is acceptable only in applications where the errors cannot have serious consequences, e.g. they may cause bad pixels that are not noticed in an animation or they may change the probability of some token in LLM inference, which may not change the actual sampled output, and even if the output is changed, it might not matter among more important causes of hallucinations.

The one way I know of for denormals to arise in real calculations is when you have a process that converges on zero. In which case, values will pass through the denormal range on the way from the normal range down to zero. And converting denormals to zero is obviously correct.

What other cases have you seen of denormals arising in real calculations?

If you compute a limit numerically, the decision that you have reached the limit must normally be taken long before there is any possibility of underflow, i.e. of generating denormals.

Typically you compute the difference between 2 adjacent terms of the convergent sequence and you decide that you have reached the limit when adding the difference to the current term to get the next does not change it (or when the relative error is smaller than some threshold). At this time the difference will still be many orders of magnitude greater than a denormal.

If the limit happens to be zero, then what you describe can happen. The programs where this can happen normally combine two different criteria to decide that the limit has been reached, i.e. that either the relative error is small enough, i.e. like I said that adding the difference to the previous term does not change it, or that the absolute error is small enough, i.e. that the difference is smaller than some threshold. The absolute error used as threshold is normally chosen based on what is physically meaningful in that problem, i.e. depending on what kind of physical quantity corresponds to the values of the terms of the convergent series.

An example of applications where the users must configure both relative errors and absolute errors, to be used as criteria of convergence, are the SPICE-like programs used to simulate electronic circuits, where the user must configure both a generally-applicable relative error, and absolute errors for different kinds of physical quantities, e.g. an absolute error for voltages and an absolute error for electric currents, which will be used, respectively, when a sequence of voltages converges towards zero or a sequence of currents converges towards zero, so that the absolute error criterion will be satisfied before the relative error criterion is satisfied.

In any case, in a correct program the attempt to find the limit should always stop before underflows become possible, so denormals should never be generated if the underflow exceptions are masked.

Denormals can appear in a lot of cases when almost equal values are subtracted, e.g. in the solution of many kinds of badly conditioned equations, but in most such cases there may exist alternative formulae that avoid underflows, i.e. the generation of denormals.

In general, when denormals appear, this signals bugs in the program, which must be investigated and fixed. The purpose of denormals is to allow the programmer to not fix the bugs, without causing catastrophic errors, like those that can happen when underflows generate null results, i.e. when "denormals are flushed to zero".

Careful programmers should nonetheless fix any bugs that generate denormals.

Ah! So let me paraphrase to make sure I understand. In your experience, the real use case of denormals is not that you want to compute with them. It's that they are a useful error signal, because they typically indicate that the equations are badly conditioned, which means the results cannot be trusted. So in that scenario, silently flushing them to zero is bad, but so is silently computing with them. What you really want is for denormals to raise an exception?

They are indeed a useful error signal, like also infinities or NaNs.

Moreover, they are much more benign than infinities or NaNs, which signal bugs that you most likely have to fix.

Denormals cause only very small errors, which may be acceptable in most applications. Therefore you may choose to ignore such bugs and not fix them, because a modified algorithm that avoids underflows may be significantly more complex than the original algorithm.

Configuring the non-standard option to flush denormals to zero is bad for two reasons. Not only it removes evidence that something bad has happened, but unlike with denormals, this can cause huge errors, even infinite errors.

When the operations are done according to the IEEE standard, every operation has a limit for the relative error and it is possible to do a numeric analysis of some computational algorithm and estimate the maximum errors that can affect its results.

When underflows generate neither exceptions nor denormals, all guarantees are removed and you can no longer predict anything about the final errors of an algorithm.

Reply on news.ycombinator.com