> [...]
> Floating-point arithmetic is hard.
I have successfully avoided FP code for most of my career. At this point, I consider the domain sophisticated enough to be an independent skill on someone's resume.
> [...]
> Floating-point arithmetic is hard.
I have successfully avoided FP code for most of my career. At this point, I consider the domain sophisticated enough to be an independent skill on someone's resume.
There is actually no better way, if you try to calculate over the reals (with computers or whatever you want), you are prone to be bitten. Once in a while there's an article about intervalar algebra on HN, those are a great opportunity to just nod positively and remember all of the flaws of intervalar algebra I got to learn on my school's physics labs. (And yeah, those flaws do fit some problems better than FP, but not all.)
Numerically unstable algorithms are a problem too but again, intuitively so if you think of the numbers as physical measurements.
These bugs are so subtle and so pervasive that its almost always cheaper to throw more hardware at the problem than it is to hire a numerical analyst. Chances are that you aren't clever enough to unit test your way out of them, either.
The trouble is that people end up using them for any non-integer ("real") numbers. It turns out that in modern times scientific calculations with measured values are not necessarily the bulk of calculations in actually written software.
In the 21st century, i don't think there's any good reason for literals like `21.2` to represent IEEE floats instead of a non-integer data representation that works more how people expect for 'exact' numbers (ie, based on decimal instead of binary arithmetic; supporting more significant digits than an IEEE float; so-called "BigDecimal"), at the cost of some performance that you can usually afford.
And yet, in every language I know, even newer ones, a decimal literal represents a float! It's just asking for trouble. IEEE float should be the 'special case' requiring special syntax or instantiation, a literal like `98.3` should get you a BigDecimal!
IEEE floats are a really clever algorithm for a time when memory was much more constrained and scientific computing was a larger portion of the universe of software. But now they ought to be a specialty tool, not the go-to for representing non-integer numbers.
There are still a lot of people doing a lot of work in which they hardly ever want a floating point number but end up using it because it's the "obvious" one that happens when you just write `4.2`, and the BigDecimal is cumbersome to use.
It's a matter of time if one doesn't know to look for numerically stable algorithms. Or if one thinks performance merits dropping stability.
https://github.com/RhysU/ar/issues/3 was an old saga in that vein.
1 - quantity2 / (quantity1 - quantity2)
... or some such thing. If quantity1 and 2 are similar, ouch!
quantity2/quantity1 - 1
or
(quantity2 - quantity1) / quantity1
with double precision and physically reasonable values.
It's been a whole field with its own patron saint for a quite a while, take a look at
Sadly, I suspect too many "computer science" courses have turned into "vocational coding" courses, and now those people are computing summary statistics on large datasets in Javascript...
A large dataset means lots of values, maybe we can assume the number of values is way bigger than any individual value. Perhaps think of McDonalds purchases nation-wide: billions of values but each value is probably less than $10.
The simplest summary statistic would be a grand total (sum). If you have a good mental model of floats, you immediately see the problem!
The mental model of floats which I use is 1) floats are not numbers, they are buckets, and 2) as you get further away from zero, the buckets get bigger.
So let’s say you are calculating the sum, and it is already at 1 billion, and the next purchase is $3.57. You take 1 billion, you add 3.57 to it, and you get... 1 billion. And this happens for all of the rest of the purchases as well.
Remember: 1 billion is not a number, it is a bucket, and it turns out that when you are that far away from zero, the size of the bucket is 64. So 3.57 is simply not big enough to reach the next bucket.
It was precisely this problem. The individual had done all data preparation/normalization in 32-bit because the model training used 32-bit on the GPU. It's a very reasonable mistake if one hasn't been exposed to floating point woes. I was pleased to see that the individual ultimately caught it when observing that 2 libraries disagreed about the mean.
Computing a 64-bit mean was enough. Compensated (i.e. Kahan) summation would have worked too.