Hacker News Reader

Comment by Tainnor | Hacker News Reader

These aren't the right lessons to learn from this issue.

Comparing floats for equality is not a good idea, not just because of the 0.1+0.2!=0.3 behaviour (which, yes, is at least consistent), but because floats - or numerical methods in general - are simply not meant to be used for exact results. If that's what you want - e.g. big integer or fixed point arithmetic - there are other types you should be using. So the takeaway is not that JS has a "glitch" or is messing up - but a misunderstanding of the guarantees of numerical methods.

Also, the suggested solution to compare actual vs. expected value by having their absolute distance be less than some epsilon is half correct, but what you really should be looking at is the relative error, i.e. the absolute error divided by the magnitude of the number.

View on HN

> but what you really should be looking at is the relative error

This actually depends on the specific problem you're talking about, and there's also the issue of forward vs. backward error. Unless you're doing something that requires additional care, unit-testing against abs(expected - actual) < 0.001 is relatively safe (and if you are, then God help you).

100% agree on the rest of the comment, requiring numerical methods to converge to a specific IEEE 754 double precision floating point is just dumb.

I'm not very experienced with this in practice and can mostly only speak to what I was taught in numerical methods, but it was my understanding that a) relative errors don't arbitrarily depend on your units and b) they seem to be better behaved in theory, or at least, it's how algorithms are typically analysed - e.g. if you multiply two numbers that both have some associated error, the relative error is the sum of the individual relative errors (plus quadratic terms which you would ignore).

1e-3 is a pretty loose error bound if you're dealing with double-precision. I usually use tolerances in the region of 1e-5 to 1e-10.

It's also better to use significant figures or a relative (as opposed to absolute as computed above) error if you're working with small values.

> Comparing floats for equality is not a good idea

This is very true, but the mention of data science is really significant here.

If you're trying to productionize a neural network or something similar, you ideally want to be able to pin down every source of noise to make sure your results are reproducible, so you can evaluate whether a change in your results reflects a change in the data, a change in the code, or just luck from your initial weights - with a big enough network, for all you know it decided that the 10th decimal place of some feature was predictive. I wouldn't be surprised if the author brings up tanh as an example specifically because it's a common activation function.

If you're pinning all your Javascript dependencies, but the native code it dynamically links to gets ripped out from under you, it could completely mislead you about what produces good results. Similarly, if you want to be able to ship a model to run client-side, it would be nice if you could just test it on Node with your Javascript dependencies pinned and be reasonably confident it'll run the same on the browser.

Of course, if you can't do that it's not the end of the world, since you can compensate by doing enough runs, and tests on the target platform, to convince yourself that it's consistent, but it's a lot nicer if things are just as deterministic as possible.

I'm not a data scientist, but

> for all you know it decided that the 10th decimal place of some feature was predictive

isn't this kind of a misfeature of the model? If it really depends that highly on a function output that's not guaranteed to be precise even in the best of cases and certainly not when you have to assume noisy input data... that doesn't seem to be particularly robust.

If you want true reproducibility you can save everything in an image/container. Lots of data science companies offer this for end-to-end reproducibility.

>> because floats - or numerical methods in general - are simply not meant to be used for exact results

But one should expect a single function does not vary in the last 5 digits as one example did. That's over 16 bits. Why should anyone use double precision if that's the kind of slop an implementation can have?

> But one should expect a single function does not vary in the last 5 digits as one example did.

Why not? The whole selling point of floats is that they're a fast approximation to real arithmetic in a bunch of useful cases. I'd personally be much more miffed by an excessively slow implementation of a transcendental than one which was off even by a large number of bits. If there's a fast solution with better precision then that's great, but low precision by itself doesn't strike me as particularly problematic.

> Why should anyone use double precision if that's the kind of slop an implementation can have?

Because by using float64 instead of float32 you can cheaply get a ton more precision even with sloppy algorithms. If there's only 16ish bits of slop you could probably get a full-precision float32 operation just by using an intermediate cast to float64 (proof needed for the algorithm in question of course).

> But one should expect a single function does not vary in the last 5 digits as one example did. That's over 16 bits.

    0.9999999999999999999999999999999
    1.0000000000000000000000000000000

Counting the number of different digits has no useful meaning. The example you cite has two results that differ by 1 part in 10 quadrillion. If you have a good reason care about the precise implementation of hyperbolic functions to that level of precision then you don't rely on whatever you happen to stumble on in some version of nodejs.

If you're talking about the tanh example, that differs in the last five bits of the mantissa. You don't want to count the number of different bits because differences in the more significant bits are worse. Count the ulps instead.

Reply on news.ycombinator.com