This wouldn't work for C/C++ because in these languages signed integer overflow is undefined behavior.
Go was designed by (among others) the father of Unix Ken Thompson, with an understanding of the mistakes of C and C++.
Another example is that Go requires explicit integer casts (disallowing implicit integer casts) to avoid what is now understood to be an enormous source of confusion and bugs in C.
You can understand Go as an improved C, designed for a world where parallel computing (e.g., dozens of CPU cores) is commonplace.
Well, that's arguable. This "mistake" could be fixed in C tomorrow without breaking the semantics of any existing C code, but notice that this hasn't been "fixed" in any of the latest C standards, so perhaps it's still there for a reason.
And the reason it hasn't been "fixed" is that compilers can optimize code better if they can assume that signed addition won't overflow.
So it's more of a trade-off rather than strictly being a disadvantage.
It's also something you can "fix" in your own code if you want to, by passing a compiler flag (-fwrapv in gcc), although arguably, at that point your code wouldn't be strictly C-standard compliant anymore. Or by using some library that handles arithmetic overflow by wrapping around, which could be implemented in standard C.
> Another example is that Go requires explicit integer casts (disallowing implicit integer casts) to avoid what is now understood to be an enormous source of confusion and bugs in C.
I agree with you on this, although forcing explicit casts also makes the code more verbose and can make it harder to understand what's going on.
I think a balanced approach is requiring explicit casts only for the "tricky" cases, i.e. when the values might become truncated, and possibly also when sign-extension might be necessary and therefore might result in something the programmer didn't expect.
But if I were to design a language I'm not sure that I would require explicit casts for e.g. promoting a uint16_t to a uint32_t...
> You can understand Go as an improved C, designed for a world where parallel computing (e.g., dozens of CPU cores) is commonplace.
That's a bit of a hot take :) Let me know when the Linux kernel starts to get rewritten in Go ;)
Everyone says that and they have no proof.
Here's an entire blog post with proof:
https://kristerw.blogspot.com/2016/02/how-undefined-signed-o...
And the blog post lists dozens of optimizations which can indeed be performed due to that decision.
The benefits of these optimizations will vary depending on the actual code and on the architecture/CPU, of course.
It will be greater for hot and tight loops that benefit from these optimizations and less important for cold parts of the code.
It will also be greater for more limited CPUs and less important for more sophisticated ones.
> Signed integer expression simplification [...]
This is easily addressed by allowing (but not requiring) implementations to keep wider intermediate results on signed overflow, to a unspecified width at least as wide as the operands.
> the index calculations are typically done using 32-bit int
No, they're done using size_t; that's what size_t is for.
> undefined overflow ensures that a[i] and a[i+1] are adjacent.
No, the fact that i (of type size_t) < length of a <= size of a in bytes <= SIZE_MAX ensures that a[i] and a[i+1] are adjacent.
(Also, note that this does not mean the optimizer is allowed to access memory at &a[i+1], since that might be across a page boundary (or even the wraparound from address ...FFF to 0), which makes adjacency less helpful than one might hope.)
> Value range calculations [...] Loop analysis and optimization [...]
Allowed but not required to retain excess bits on signed overflow.
> This is easily addressed by allowing (but not requiring) implementations to keep wider intermediate results on signed overflow, to a unspecified width at least as wide as the operands.
> > Value range calculations [...] Loop analysis and optimization [...]
> Allowed but not required to retain excess bits on signed overflow.
What does that even mean? That the code would have different behavior with a different compiler, different optimizations or when you slightly change it (depending on whether the compiler chooses to keep wider intermediate results or not)?
If understand you correctly, then depending on the values of x and y, `(x+1)<(y+3)` would have a different result than `(x+a)<(y+b)` when a=1 and b=3, because in the first case you would be simplifying the expression but in the second case you couldn't.
That would be quite surprising, to say the least.
> No, they're done using size_t; that's what size_t is for.
for (int i = 0; i < 256; i++)
a[i] = b[i] + c[i];
Also, when doing arithmetic with indices, you might need to represent a negative index, so `size_t` wouldn't work. You'd need to use `ssize_t`, which is a signed integer which would benefit from these optimizations.
But even then you might use an `int` if you know the arithmetic result will fit.
> No, the fact that i (of type size_t) < length of a <= size of a in bytes <= SIZE_MAX ensures that a[i] and a[i+1] are adjacent.
Not if `i` is a signed integer, say, `int` or `int8_t`. Which is the point of the optimization.
If x > TYPEOFX_MAX or y > TYPEOFY_MAX-2, then this can already happen with the more (maximally) vague "signed overflow is completely undefined" policy; wider intermediates just mean that code is not allowed to do other things, like crash or make demons fly out of your nose.
If x+[a or 1] and y+[3 or b] don't overflow, then computing them in a wider signed integer type has no effect, since the same values are produced as in the original type.
More generally, retained overflow bits / wider intermediates (instead of undefined behaviour) mean that when you would have gotten undefined behaviour due to integer overflow, you instead get a partially-undefined value with a smaller blast radius (and hence less opprotunity for a technically-standards-conformant compiler to insert security vulnerabilities or other insidious bugs). In cases where you would not have gotten undefined behaviour, there is no signed integer overflow, so the values you get are not partially-undefined, and work the same way as in the signed-overflow-is-undefined-behaviour model. ... you know what; table:
signed overflow is \ no overflow unsigned overflow signed overflow
undefined behaviour: correct result truncated mod 2^n your sinus cavity is a demon roost
wide intermediates: correct result truncated mod 2^n one of a predictable few probably-incorrect resultsYes, but saying "signed overflow is completely undefined" simply means that you are not allowed to do that, so this is a very well-defined policy and as an experienced programmer you know what to expect and what code patterns to avoid (hopefully).
If you say "signed overflow is allowed" but then your code behaves nondeterministically (i.e. giving different results when signed overflow happens, depending on which compiler and optimization level you're using or exact code you've written or slightly changed), I would argue that would actually be more surprising for an experienced programmer, not less!
It would make such signed overflow bugs even harder to detect and fix! As it would work just fine for some cases (or when certain optimizations are applied or not, or when you use a certain compiler version or Linux distro) but then it would completely break in a slightly different configuration or if you slightly changed the code.
And it would prevent tools like UBSan from working to detect such bugs because some code would actually be correct and rely on the signed overflow behavior that you've defined, so you couldn't just warn the programmer that a signed overflow happened, as that would generate a bunch of false alarms (especially when such signed-overflow-relying code was part of widely used libraries).
> More generally, retained overflow bits / wider intermediates (instead of undefined behaviour) mean that when you would have gotten undefined behaviour due to integer overflow, you instead get a partially-undefined value with a smaller blast radius (and hence less opprotunity for a technically-standards-conformant compiler to insert security vulnerabilities or other insidious bugs).
C compilers are already allowed to do what you say, currently. But I'm not sure that relying on that behavior would be a good idea :)
I think it's preferable that the C standard says that you are not allowed to overflow signed integers, because otherwise the subtlety of what happens on signed overflow would be lost on most programmers and it would be very hard to catch such bugs, especially due to code behaving differently on slightly different configurations (hello, heisenbugs!) and also because bug-detection tools couldn't flag signed overflows as invalid anymore.
No, saying "signed overflow is a compile-time error" means you're not allowed to do that. Saying "signed overflow is completely undefined" means you are allowed to do that, but it will blow up in your face (or, more likely, your users' faces) with no warning, potentially long after the original erroneous code change that introduced it.
> As it would work just fine for some cases (or when certain optimizations are applied or not, or when you use a certain compiler version or Linux distro) but then it would completely break in a slightly different configuration or if you slightly changed the code.
That sentence is literally indistinguishable from a verbatim quote about the problems with undefined behaviour. Narrowing the scope of possible consequences of signed overflow from "anything whatsoever" to "the kind of problems that are kind of reasonable to have as a result of optimization" is a strict improvement.
> And it would prevent tools like UBSan from working
> bug-detection tools couldn't flag signed overflows as invalid
The standard says `if(x = 5)` is well-defined, but that doesn't stop every competently-designed (non-minimal, correctly configured[0]) compiler from spitting out something to the effect of "error: assignment used as truth value".
0: Arguably a fully competently designed compiler would require you to actually ask for -Wno-error rather than having it be the default, but backward compatibility prevents changing that after the fact, and it would require configuring -Wall-v1.2.3 (so build scripts didn't break) anyway.
> No, saying "signed overflow is a compile-time error" means you're not allowed to do that.
In the vast majority of cases it's not possible to statically determine if signed overflow will occur, so compilers can't do that. I'm sure they would do it, if it were possible.
> Saying "signed overflow is completely undefined" means you are allowed to do that, but it will blow up in your face
No, it does not mean that. You're not allowed to do signed overflow in standard-compliant C, period.
You're allowed to do signed arithmetic, but the arithmetic is not allowed to overflow. You can write code that overflows, but it will not have defined semantics (because that's what the standard says).
And the compiler cannot enforce or emit a warning when overflow occurs because in the general case it's not possible to statically determine if it will occur.
But if the compiler can determine it, then it will emit a warning (at least with -Wall, I think).
And if you pass the `-ftrapv` flag to GCC (and clang, probably), then your code will deterministically fail at runtime if you do signed overflow, but for performance reasons this is not required by the standard.
> > As it would work just fine for some cases (or when certain optimizations are applied or not, or when you use a certain compiler version or Linux distro) but then it would completely break in a slightly different configuration or if you slightly changed the code.
> That sentence is literally indistinguishable from a verbatim quote about the problems with undefined behaviour. Narrowing the scope of possible consequences of signed overflow from "anything whatsoever" to "the kind of problems that are kind of reasonable to have as a result of optimization" is a strict improvement.
No, because experienced programmers don't expect signed overflow to work, because it's not allowed. Such bad code would be caught if you enable UBsan. But if the C standard would require what you propose, then UBsan could not fail when a signed overflow occurs, as that could be a false positive (and therefore make such signed-overflow detection useless).
If you allow signed overflows then you have to define the semantics. And nondeterministic semantics in arithmetic is prone to result in well-defined but buggy code, while in this case also preventing bug-detection tools from being reliable.
That said, you could conceivably implement such semantics in a compiler, which you could enable with some flag, like for example, -fwrapv causes signed overflow to wraparound and -ftrapv causes signed overflow to fail with an exception.
So you could implement a compiler flag which does what you want, today, and get all the benefits that you're proposing.
You could even enable it by default, because the C standard allows the behavior that you're proposing, so you would not be breaking any existing code.
And this would also mean that existing and future C code would still be C-standard compliant (as long as it doesn't rely on that behavior).
But making the C standard require that behavior means that there will be well-defined, standard-compliant code that will rely on those weird semantics when signed overflow occurs, and that's a really bad idea.
> > bug-detection tools couldn't flag signed overflows as invalid
> The standard says `if(x = 5)` is well-defined, but that doesn't stop every competently-designed (non-minimal, correctly configured[0]) compiler from spitting out something to the effect of "error: assignment used as truth value".
The big difference in this case is that such errors are easy to determine at compile-time. No compiler would cause such code to fail at run-time, because it would lead to unexpected and unusual program crashes, which would result in both users and programmers to be mad (especially if the code is correct). But for signed overflows, it's not possible to implement a similar compile-time error.
As another example, AddressSanitizer is competently-designed but as soon as you enable `strict_string_checks` you will run into false positives if your code stores strings by keeping a pointer and a length, rather than forcing them to be NULL-terminated, so that flag can be completely useless (and for my current project, it is).
Which is why I'm guessing it's disabled by default. Which means almost nobody uses that flag.
This happens because strings are not actually required to be NULL-terminated in C, even though most people use them that way. So there is code out there (including mine) that relies on strings not always being NULL terminated, and this has well-defined semantics in C.
But of course, as soon as that happens, then you can't rely on the tool to be useful anymore, because there is perfectly fine code which relies on the well-defined semantics.
Note that in this case, "strict_string_checks" is a run-time check, like detection of signed overflows would have to be.
For 32 bit systems using 64bit instead of size_t would similarly solve the problem.
Conceptually, a 64-bit kernel today could allow your program to allocate (almost) the entire 64-bit address space, assuming it does memory overcommit (like Linux) and/or uses some kind of memory compression (like Linux supports as well).
There might be some MMU limitations on today's mainstream systems, but this doesn't mean that all 64-bit systems have those limitations or that those limitations will remain there in the future.
So your code would break as soon as a new system comes along without those limitations.
Also, this would be even more true if the code and stack would be stored in different address spaces, as theoretically that would even allow you to allocate the entire address space, I think.
A 48-bit index to an array can represent >240TBytes of RAM minimum - if your records are > 1 byte, you have significantly higher storage requirements. The largest system I could find that’s ever been built was a prototype that has ~160TiB of RAM [1]. Also remember. To make the algorithm incorrect, the sum of two numbers has to exceed 64bits - that means you’d need >63-bits of byte-addressable space. That just simply isn’t happening.
Now of course you might be searching through offline storage. 2^63 bits is ~9 exabytes of an array where each element is 1 byte. Note that now we’re talking scales of about about the aggregate total storage capacity of a public hyperscaled cloud. Your binary search simply won’t even finish.
So sure. You’re technically right except you’d never find the bug on any system that your algorithm would ever run on for the foreseeable future, so does it even matter?
As an aside, at the point where you’re talking about 48-bits worth of addressable bytes you’re searching, you’re choosing a different algorithm because a single lookup is going to take on the order of hours to complete. 63-bits is going to take ~27 years iff you can sustain 20gib/s for comparing the keys (sure binary search is logarithmic but then you’re not going to be hitting 20gib/s). Remember - data doesn’t come presorted either so simply getting all that data into a linearly sorted data structure is similarly impractical.
"Today" being the important part. That could change tomorrow. I could implement a 64-bit CPU right now that would support it (on an FPGA). It's not an inherent limitation, it's just an optimization that current CPUs do because we don't need to use the full 64-bit address space, usually.
Also, address space doesn't necessarily correspond 1-to-1 with how much memory there is.
For example, according to the AddressSanitizer whitepaper, it dedicates 1/8th of the virtual address space to its shadow memory. It doesn't mean that you need to have 2 exabytes of addressable storage to use AddressSanitizer, or that it reads or writes to all that space.
As I said, memory overcommit and memory compression (and also page mapping in general, as well as memory mapping storage and storage compression and storage virtualization, etc) allow you to address significantly more memory (almost infinitely more) than what you actually have.
There are other tricks with memory, page mapping and pointers which could break your code if it's not standards-compliant. This could happen for security reasons or because of new compiler or kernel optimizations or new features.
So I agree that this isn't a problem right now, unless you're doing something very esoteric, but if you want to have standards-compliant code and be more future-proof then you cannot rely on that.
There is also the point that the Go code that we're discussing has nothing to do with arrays, memory or address spaces, because it's a generic binary search function that works for any function "f" passed as an argument.
For example, it can be used to do a binary search for finding the zero of a mathematical function (i.e. for finding which value of `x` results in `y` becoming zero in the equation `y=f(x)`) and this has nothing to do with address spaces.
You’re hand waving away way too much complexity. Please do build this system. Keep in mind that addressing 63bits of memory with huge tables on will use up > 2 Tera worth of PTEs which translate to what, 16 Terabit worth of memory? This is simply an order of magnitude more than dedicated machines ship with. You’re certainly not getting an FPGA with that.
> For example, according to the AddressSanitizer whitepaper, it dedicates 1/8th of the virtual address space to its shadow memory. It doesn't mean that you need to have 2 exabytes of addressable storage to use AddressSanitizer, or that it reads or writes to all that space.
I think you’re failing to appreciate how large 2^63 bytes is.
> As I said, memory overcommit and memory compression (and also page mapping in general, as well as memory mapping storage and storage compression and storage virtualization, etc) allow you to address significantly more memory (almost infinitely more) than what you actually have.
See point above. Such a system is just not likely to exist in your lifetime.
> but if you want to have standards-compliant code and be more future-proof then you cannot rely on that.
All code has a shelf life. What’s the date you’re working on here? I’m willing to bet it’s not an issue by the end of this century.
The page table is itself stored in virtual memory, is a tree structure and it can be fully sparse, i.e. you only need to populate the PTEs that you use, basically.
Keep in mind, as long as you enable memory overcommit or use MAP_NORESERVE in mmap, you can allocate 127 TiB (~2^47 bytes) worth of virtual address space on Linux x86-64, today, at zero cost. With 4K pages!
In fact, I just did that on my laptop and memory usage has remained exactly as it was before.
And on POWER10 you can map 2^51 bytes of virtual address space today, also at zero cost.
> I think you’re failing to appreciate how large 2^63 bytes is.
No, I do appreciate it. It's 65536 times larger than the maximum address space you can allocate today on Linux x86-64 at zero cost. With 4K pages. Or a factor of 2048 larger than POWER10 can do today, also at zero cost.
In fact, with 1G HugePages, the maximum theoretical number of PTEs needed for 2^64 bytes of address space would be LESS than the number of PTEs needed for the 2^47 bytes you can allocate today, on Linux x86-64, with 4K pages, at zero cost (which I just did, on my laptop).
The maximum amount of virtual address space you can allocate is only limited by how many bits the CPU and MMU are designed to address.
If you don't believe me, you can try it yourself:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main() {
// 127 TiB
size_t size = 127ULL * 1024 * 1024 * 1024 * 1024;
printf("allocating %zu bytes of virtual address space...\n", size);
void *p = malloc(size);
if (p == NULL) {
perror("malloc");
exit(1);
}
printf("success: %p\n", p);
sleep(3600);
}
$ gcc -o alloc alloc.c -Wall
$ ./alloc
allocating 139637976727552 bytes of virtual address space...
success: 0xf29bb2a010
Just to be clear, even if a PTE entry was just 1 pointer long (it's not), covering 63 bits of address space with 1 GiB PTEs would require >73 GiB just for the page tables. And those page tables ARE getting materialized if you're doing a binary search over that much data.
I'm not as imaginative in you to see a world in which you can sparsely map in 2^63 elements (9 exabytes if 1 byte per element) on one CPU and then the problem you're solving is a binary search through that data which is going to cause about log(n) to be mapped in to satisfy the search. 1 exabyte is probably the amount of RAM that Google has collectively worldwide. Now sure, maybe you're talking about mapping files on disk but again. 1 exabyte is a shit ton. It's probably several clusters worth of machines for storage. And even with 1 GiB pages, you're talking about 1 billion PTEs total and each lookup is going to need to materialize ~9 PTEs to search. And all of that is again a moot point because no CPU like that exists or will exist any time soon.
~~C does not guarantee a 64 bit type exists.~~ This is not really correct these days.
Isn't (signed/unsigned) 'long long int' mandatory since C99?
It says: 'There are five standard signed integer types, designated as signed char, short int, int, long int, and long long int'.
My cursory search for 'long long' in the standard didn't find anything about it being optional...
And technically speaking, I think C doesn't guarantee that 'size_t' is at least as large as a 'signed int' (even though this is true on all platforms that I know of), so your approach would fail if that weren't the case. Although, you could use 'ssize_t' instead of 'int', or 'unsigned int' instead of 'size_t' to fix that.
> The real reason it doesn't work in C is because a memory region can be large enough to still overflow in that case.
The Go code we are discussing has nothing to do with memory regions, it's a generic binary search function, so it can be used for e.g. bisecting git commits. It doesn't require the calling function to use arrays.
Although yes, if the calling code were trying to do a binary search on an array, conceptually it could fail, but in that case you could argue the bug would be in the calling function, because it would be trying to pass the array length into a binary search function which only accepts an `int` or `ssize_t` function parameter, which could result in the array length being truncated. But strictly speaking, this would not be an arithmetic overflow issue.
That said, I would just fix the code so that it works for the full 'size_t' range, since the most common use case of a binary search function is indeed to do searches on arrays. In that case, the Go approach wouldn't work indeed.
That doesn't matter, because size_t is large enough to hold any array index (that's kind of[0] the defining property of size_t), so any array index in a signed int can be safely converted to size_t. The real problem is that (using 16-bit size_t for illustrative purposes) if you have, say, x = (size_t)40000 and y = (size_t)50000 into a 60000-element array, x+y = (size_t)90000 = (size_t)24464, which means (x+y)/2 = 12232, which is the completely wrong array element.
0: Technically, size_t is large enough to hold any object size, but array elements can't be smaller than char (sizeof can't be less than 1), so a array can't have more elements than it's sizeof.
> That doesn't matter, because size_t is large enough to hold any array index (that's kind of[0] the defining property of size_t), so any array index in a signed int can be safely converted to size_t.
Well, the Go code we're discussing has nothing to do with arrays or array indices, so `size_t` doesn't help here.
Go look at the code :) It's a generic function for doing binary search, which accepts an `int` as a function argument, specifying the search size.
The code is then doing:
h := int(uint(i+j) >> 1) // avoid overflow when computing h
That's the point I was making.