New zine: How integers and floats work
jvns.ca
jvns.ca
Yeah, the actual implementation is fairly different, but that’s just computer gibberish. Most of us no longer need to think of strings as an offset into memory to an array of characters that happen to be interpretable as a group, but that’s really what it is despite any abstractions.
The real mystery is why most modern languages continue to use floats and integers as the default numeric types rather than some sort of BigDecimal representation, given that precision and programmer intuitiveness trumps raw performance in the vast majority of cases.
Floating point are used because quite simply they are the only way to do it that has wide support that can be reused.
A good fixed width or Decimal implementation that could be reused and implement all the maths you want for Reals are... Rare.
The exception is Java and JavaScript mostly. But JS has Bigint now. Just not as a default.
I’m not saying get rid of these distinctions entirely. But where they matter, for performance reasons or whatever, they should be opt-in rather than the default.
Many floats (such as 14.7) are not even represented exactly, float operations have a large number of issues.
If you're wondering about all this, see:
What Every Computer Scientist Should Know About Floating-Point Arithmetic
~ David Goldberg, March, 1991 issue of Computing Surveys
https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.h...
(and elsewhere)
I'm sorry, my apologies.
Are you saying that IEEE754 does exactly represent 14.7 ?
> Speaking as someone working in numerical analysis and scientific computing
You're hardly the only Scot in Glasgow.
I feel certain a few here abouts have arc'd through PDP-11's, Cyber 205's, and landed in modern centres such as Pawsey.
I have no problem with languages having first-class support for big integers, arbitrary precision floating point numbers, arbitrary precision rationals, etc. The more representations the merrier, provided they have an efficient implementation.
Having programming languages where floating point types are first class, unboxed types is extremely important for anyone who writes software which crunches numbers. I would say this is more important than providing training wheels for inexperienced programmers.
In 5 seconds or less, can you say whether 1.0/4 will be 0.25 or 0 for every language you use regularly? Most developers can't, and most languages make statements like that trivial (and worse, it far more often shows up as "x/y" for x and y of types that need to be looked up to know what the evaluated type will be). There's a reason C++, for example, has the MISRA rules about avoiding implicit coercions (because even though what happens is [generally] well-defined, enough code in the wild has been broken for developers mis-remembering or mis-applying the rules that explicit casts are highly recommended for high-reliability code).
The languages I work with most often in decreasing order of frequency are: C, Python, C++, Fortran, MATLAB, and Julia. I don’t regularly cause problems for myself by running afoul of implicit casting rules, but I also spend all day writing numerical software. I can see that it can cause problems for other people who don’t spend as much time dealing with IEEE754.
There are definitely days where I feel like an OCaml-like level of casting pedantry would be nice, and other days I feel like it would be exhausting and make my code harder to read. I’ve never written a big piece of code in a language that required explicit casts for everything. I’m open to it.
I would say that unsigned integers cause me the most problems of any numeric type, but not so many problems that I’m looking to change languages. If I worked in a different field it’d be a different story I’m sure.
What languages don't evaluate 1.0/4 to 0.25? I checked OCaml (exception), node (0.25), Lua (0.25), Swift (0.25), sbcl (0.25), ... and then I got bored.
To me the expected behavior should be something like: for a binop between a float and an int, promote the int to float and do the floating point operation. Obviously there are corner cases here...
Pedantic: You are aware that 754 also specifies a decimal model?
It turns out that the real world is amazingly inexact, to the point that the inaccuracies of floating point just never even show up.
If it makes you feel better, with our current system, 1/256 is represented exactly. Using relatively few bits, all told. :D
This isn't without drawback, of course.
Floating point numbers are equivalent to rationals with their base as the denominator I. base-2 rationals with denominators that are powers of 2 will have termination expansions.
Useful for storage but not arithmetic as Fraction(147/10) has 10 in the denominator and 5 is a prime factor that isn't in the base.
BPD, or Binary Packed Decimal also allows for the storage of 14.7
But once again not useful for arithmetic.
Accumulated errors, catastrophic cancelation, etc are real problems too.
In decimal, 1/3 can be stored exactly but the expansion is 3 33333...
Because 3 is prime and not a prime factor of 10.
Had we decided on something rather than 10 for our base it may have worked out for computers better, but it is what it is.
The rationals are nice, being the only countable, totally ordered sets. But producing a computer circuit that can perform arithmetic on them isn't practical as far as I know without using fixed radix and the associated problems with non-terminating expansions.
Support for exact equality between floating point numbers is a perfectly useful and valid concept and should *not* be removed. You can easily design a simulation such that if you run it twice, it will give exactly the same result bitwise. This can be utterly invaluable when debugging complicated numerical software.
I admit removing language support for == of floats is a bit of a wild idea, but it is conceptually sound, if not practical. I'd support keeping it in a test/debug package.
Tho if you are looking at bitwise patterns, then I find memcmp or its equivalents to be needed, to handle Nan != NaN.
To your point, there are plenty of situations where a floating point number will be initialized to a magic constant (0, 1, infinity, so many others...). Beyond initialization, there are also loads of functions which frequently yield floating point constants which can reliably be tested for equality, like max(0, x). There are no error bars here. Exact floating point equality can usefully tell us things like "has this number changed", or more generally "has this special case occurred"/"have we reached a particular state".
Removing floating point == isn't conceptually sound, and it isn't practical.
What are the cases you are mentioning? Surely something more complex than a float for a loop variable? When the error is less than a threshold, when the sign changes, when the number of objects 8n a partition is one, I can’t think of a calculated float that would be a test for something with equality.
IMO, there is a huge qualitative difference between 15 and 14.7; 15 is an integer, while 14.7 isn’t.
I also think most people would know that instinctively. Reason? I think most people learn arithmetic progressing more or less along these lines:
- positive integers
- non-negative integers (i.e. learning and understanding the concept of ‘zero’)
- non-negative rationals
- all rationals
- decimal fractions
- all reals
Also, as sfpotter already said, switching to doubles only isn’t a panacea. Neither are bigints (won’t help representing the result of ⅓ exactly, for example), rationals (won’t help representing the typical hypothenuse of a triangle exactly, for example), or radicals (won’t help computing the area of a unit circle exactly)
A CAS would help, but by then, you’re giving up lots and lots of performance.
(that doesn’t mean I am a fan of languages that silently round integer division. Silently moving to bigints and rationals IMO can be a good choice, but silently moving from an exact type to an inexact one IMO never is)
Yes and no. The fact that one is an integer and one isn’t only really matters when considering them in the context of a set, e.g. when counting. But there’s nothing special about integers that affects how we use them relative to decimals. Multiply both of them by 10, and you’re back to integers. Divide them both by 10, and you’ve got decimals. It’s perfectly reasonably for all mathematical purposes to consider 15 and 15.0 to be equivalent to each other. This isn’t the case with computers, and that has nothing to do with mathematics.
Speak for yourself :)
I've had success discussing floats with compute engineers by framing it like a really large fixed point number with all zeroes, except a small bit range. The location and value of the bit range are set by the exponent and mantissa, respectively. This makes precision issues and things like denormals straight forward to discuss.
If there was hardware support for 128-bit floating point, we could just use that: 113 bits of precision and 15 bits of exponent is enough for nearly any number you would ever want to store or process. We'd just need to agree on a sensible rounding scheme for turning those into array indexes.
Also I don't think it would matter if a language like Python would switch to BigDecimals as the default option. Python does such a ridiculous large amount of stuff unrelated to the actual math, that bit of lost performance won't break the camel's back.
In case anyone hasn't seen it yet, if you print a double with enough fractional values, you'll eventually get its "true" value (because 2 divides 10).
>>> print('%.60f' % 0.1)
0.100000000000000005551115123125782702118158340454101562500000
Here are the "true" values of 0.2, 0.3, and their sum:
>>> print('%.60f' % 0.2)
0.200000000000000011102230246251565404236316680908203125000000
>>> print('%.60f' % 0.3)
0.299999999999999988897769753748434595763683319091796875000000
>>> print('%.60f' % (0.1 + 0.2))
0.300000000000000044408920985006261616945266723632812500000000
>>> print('%.60f' % math.nextafter(0.3, 100))
0.300000000000000044408920985006261616945266723632812500000000
The algorithm to do this efficiently is quite interesting, too.
If you round to the nearest, like, 10E-8 (maybe relative if your numbers being printed have massive range) it should be basically fine with doubles.
Unless you're displaying an unreasonable number of places to the right of the point, it's 3.
Nothing - *nothing* - needs more than about four or five significant digits.
So if you have a million in the bank, you would be fine with a 100$ rounding error on each transaction?
You could store in 1/100th of a penny, I suppose? I'm not entirely sure that will work out for all calculations.
Come to think of it, it would be a cool setting on game engines “8/16/32/64 bit floats”.
One of the nice things is that it will be faster than the arbitrary precision libs and the sum of any two numbers will be representable.
But arbitrary precision libraries are incredibly slow. Compared to radix 2 floats.
Note that it won't ever be the default, as the standard imprecise but deterministic floats are an important feature for floats.
Even with a piece of paper of arbitrary length, the real numbers are uncomputable almost everywhere and even if they weren't you still need to keep things in the countable infinity sets to be semi-decidable.
We will see if decimal types, which require 4 bits for the radix become more common now that c/cpp support them but that may be optimistic.
Are you referring to TR 24732 (at least for C)? Is that going to be in C23?
TR 24732 was withdrawn.
https://en.cppreference.com/w/c/23
Note the link to paper n2341
https://open-std.org/JTC1/SC22/WG14/www/docs/n2341.pdf
C++ is going to be a library for now I think, but haven't looked recently.
It's also strange that people use floats to represent 3D coordinates. Why should coordinates closer to the origin of the scene have more precision than those further away?
I also get that this is convenient, but the point is that it is not how a purist would do it.
(Single precision) float-induced rounding at large offsets would introduce a lot of jitter to these calculations, which looks and feels very janky.
As it stands right now the typically cited value for the distance to Alpha Centauri doesn't take the position of the earth relative to the sun into account. We live with this imprecision. And in the post office example, a conventional address would be problematic because a person might move during the 8 year round trip between the stars.
For example 0.01 + 2.01 can already be written like ((100*0.01) + (100*2.01)) / 100 which seems trivial to implement in the language.
with something like 1000000000000000.0000001 + 1000000000000002.0000001 it seems to me that one could split the operation into 2?
One could even discard 2 numbers worth of accuracy and have a representation of 1/3 and 2/3. I'm not that attached to 4294967295 Having 1/3 would be more useful.
Why would we call an implementation correct if it produces the wrong result?
And we use floats for much the same reason as you would use scientific notation. It is faster and, at large, "significant digits" don't go away just because you can write more digits.
Appeals to intuition are tough. Intuition is not some intrinsic property of the number system. It is a learned property. There is really no reason we couldn't be better at teaching it.
And with an exponent bias which isn't part of what anyone learns as basic scientific notation. Used for reasons of efficiency that don't have anything to do with the efficiency humans have in reading the exponent of scientific notation.
It's like how a theremin is basically just a monophonic piano that ramps smoothly between notes, floating free in space, scaled and rotated vertically 90-degrees, with one's entire hand being used as a single finger.
Indeed, for imperial units, this would have made a ton of sense? 1/16th of an inch would directly be .1 (1*16^-1), after all. Would be better if we used 16 symbols for counting, of course, such that we didn't mix decimal in that notation.
This, sadly, quickly devolves into debates about metric versus imperial. :(
No. The number of ratios to 16 units is 2 to a few dozens.
You have 12in to a foot, 3 ft to a yard, 1760 yd to a mile (because 80 chains or 22 yards).
A nautical mile is 10 cables, of 100 fathoms each.
640 acres to a square mile.
20 fl oz to a pint, 2 pt to a quart, 4 qt to a gallon (8 pt to a gal, 160 oz to a gal).
There's 16 drachm to an ounce and 16 oz to a pound. Sadly there's then 14 pounds to a stone, 8 st to a hundredweight unless you have abandoned stones and have 100-lb hundredweight because customary units are dumb, which then gives you a different ton of 20 cwt depending on, well, your cwt.
> 1/16th of an inch would directly be .1 (1*16^-1), after all.
Shame that would be useful for just about nothing else.
> This, sadly, quickly devolves into debates about metric versus imperial. :(
I wonder why...
That said, arbitrary numbers are arbitrary was ultimately my point. "base 10" is very inconvenient for the old imperial units, so we moved heavily to metric. And that ultimately makes sense, especially with the path dependence on how we got there. Converting between units is something folks don't like doing, so we moved a long way to turning conversions into radix shifts. Even if the old units each had individual advantages at different scales. (Consider, nautical miles are still the de facto unit in shipping, no?)
It is beyond amusing how base 10 does not work for computers that well, such that 2k of data is typically not 2000. :D
Edit: I am curious on what the argument for 10 is in scientific notation that isn't path dependent on us using decimal? Or are folks just reading a criticism of metric in my post? (Because I can easily throw in more criticisms of imperial, if that helps. :D )
The measuring stick follows the numeral system, not the other way around. And while base 2 has had some historical success, it wasn't much compared to 10, 20, 60, or 12.
> Marking it down to tenths or hundredths is surprisingly hard.
There's decimal rulers from the indus valley civilisation, it clearly was not sufficiently hard.
> Is why I picked that example, and nothing more.
From what you're saying you apparently pick that example every time and it goes wrong every time, for completely obvious reasons. Maybe you should consider changing example?
> I am curious on what the argument for 10 is in scientific notation that isn't path dependent on us using decimal?
Why would there be? We use base 10 numerals, our notation is base 10, therefore a convenient, natural, and uniform system of units would be in base 10.
Base 2 is of little help outside of length, and it's hardly critical there either.
> Or are folks just reading a criticism of metric in my post?
Well yes, 2 out of 3 paragraphs are about imperial and metric and your comment is at best tangential to the one it replies to.
This seems the case for all measurements. Used to, you wouldn't necessarily combine two units that often.
Evidence of the indus valley seems an odd choice? The images I'm finding online about them did exactly as I said, subdivide by half until happy with precision. Indeed, their "lower units of measure" were binary?
Is it?
I suppose if that stick is going to be the standard you could get a rope of the same length and fold it in half enough times.
Of course you could always make a shorter stick the standard and then find a measuring stick >= 10 times longer and add marks to the longer stick...
Edit: And yes, multiplying by basically any number is somewhat easier going up. It is specifically the down measuring that is harder to do.
Because you can trivially count to twelve on one hand. 12 joints times 4 fingers getting you to 69.
One The first few colossally abundant numbers: 2, 6, 12, 60, 120, 360 would have been ideal, with 12 most practical in hindsight. But in the golden age of India, decimal math is what they were working on and as they produced some important advancements to mathematics it won out except in areas where abundance matters like degrees, minutes, seconds.
IBM 360s supported radix 16, but radix 10, due to decimalized standards was more popular.
IEEE 754 has supported decimal floats for decades now, but as it required 4 bits, it wasn't implemented on microcomputers.
Deterministic errors with less wafer space fit most needs.
I am curious what the "4 bits" you are referencing? Seems a small number of bits, even for the time. (This is not to be confused with the 4 bit per digit of BCD, which adds up per digit encoded such that you get a lot of waste rather rapidly. Or, do I have that wrong, as well?)
Luckily, you don't need to understand the exponent bias to understand floating point. All you need to know is that you have for instance 23 bits of mantissa (plus the implicit one), an exponent with range from -126 to +127 (plus the special values -127 and +128), and a sign bit. The exact details of how the exponent is encoded in the 8 bits reserved to it are only important when you're playing tricks with the floating point representation (other than "all zero bits is +0.0", which is often important when zeroing a struct with memset).
Edit: I should say that I realize this isn't the actual exponent bias. I do think this is what would trip up most people, though?
The printf/scanf specifier, by the way, is `%a`.
Granted, I don't imagine this comes in necessary that often.
BigDecimal actually doesn't fix anything. The underlying problem is that decimal representations are rarely finite--in base 10, 1 / 3 has to get rounded to 0.333, so the same problem comes up as 1 / 3 * 3 = 0.999 ≠ 1. If you want to fix that, you have to move to rational representations, but then you get flummoxed by things like sqrt(2) or atan(1/10) or pi.
No, the thing is that 0.1 + 0.2 is 0.3 in double arithmetic. The result differs from the closest representation to 0.3 only a tiny bit--like measuring the distance between LA and NY and being off by a few atoms. The problem is that == is checking for equality of all bits rather than a tolerance, and that string printing is using rules that it prints the shortest base-10 representation that rounds to that extra floating-point value, both of which exaggerate the importance of the last few bits. So the real mystery is why no one's designed a language that makes floating-point comparisons default to relative tolerance of 10^-10 or so.
Bc the very next step is a footgun: "the function said these two values are equal but they're not."
Implementing a big decimal with feature-parity with floats is going to be a big pain.
Anyway, what about irrational numbers?
Python does big ints by default. Giving up at E-1 seems like a reasonably pragmatic but arbitrary choice.
This got me thinking, what are some things that DON’T change that are worth learning? HN I think is mostly focused on the latest things that change but there’s lot of value in focusing on things that don’t change.
Database normalization (1NF, 3NF, BCNF, etc) is worth knowing about when designing any kind of datastore. At least consider the tradeoffs of breaking BCNF. Almost all work on this was done in the 70s.
Linear algebra always turns up in unexpected places. Traditionally you would see it mostly in anything even vaguely related to computer graphics, but now it turns out to be quite useful for ML as well. Mostly stuff from the late 1800s, some a couple millennia older.
TCP/IP is mostly unchanged since the 70s, ARP is from the 80s, and HTTP1.0 is from the 90s. Combine those 4 and you have a good mental model of how the network stack works, even if HTTP3.0 is a bit more fancy.
Linux and Windows are quite different from each other, but the broad strokes of each hasn't changed much in the last two decades. For example knowing how they manage memory or how their respective thread schedulers work is potentially useful (though Linux thread schedulers do change from time to time. Windows seems to be happy with the architecture of theirs though).
Backus–Naur Form is an incredibly useful notation for describing context-free grammars. Stuff like valid syntax of a programming language, or how an address can look like. It's from the late 50s, though predecessors reach back much further. Extended Backus–Naur Form from the 90s is probably what you want to use today, it has an ISO standard and a great wikipedia page.
[0] https://jvns.ca/
Because it is clearly an ad? It explains what will you get and at what price. It does not explain "how Integers and Floats work". They want you to buy the zine, which is what they are advertising.
> Julia Evans has written a bunch of such zines in the past as well and they are quite informative
Good for them. Why do you think this makes it not an ad?
I don’t like arguing by analogy, but I don’t have a better way to make my point—-what if instead, old lady paid for a cpu that was clocked at 3.2ghz, and instead the cpu was hardware throttled down to 2.8ghz.
I expect when I purchase an item, all the features I paid for operate correctly—-even if I never expect to use all of them or use them to their full potential.
Julia has become a really, really serious pro over the years, but never lost the passion for blogging the journey. In terms of mostly self-taught hackers, an iconic legends like jwz or something at this point.
Keep it up another decade!
https://ciechanow.ski/exposing-floating-point/
and his companion website