Dereferencing an invalid pointer doesn't do "whatever this CPU does," it does completely unpredictable things. I think that's part of the problem (to the extent it's a problem)
Whether this is a good or bad thing is of course a legitimate (and good!) question, but for writing C today that's how the language is specced and a reality the programmer needs to take care to avoid, just like a bunch of other things that C leaves to the programmer like remembering to clean up allocated resources when they're no longer needed.
Any time your are dealing with data from real, physical sensors or third-party APIs, this is an impossible guarantee to give - they could literslly break.
C is an absolute minefield of undefined behavior, but let's be accurate about the things it does wrong.
C23 somewhat improves the situation with <stdckdint.h>, a standardized version of GCC’s __builtin_add_overflow and friends. That has ergonomics issues too due to its verbosity, but at least it’s hard to screw up.
if (x < 0 || y < 0) return -1;
return x * y;
if (x < 0 || y < 0) return -1;
if (x * y < 0) return -2;
return x*y;
One way to implement this check in GCC where the compiler will respect it would be:
if (x < 0 || y < 0) return -1;
int z;
if (__builtin_smul_overflow(x, y, &z)) return -2;
return z;Sure, but it didn't have to be like this. They could have said it is unspecified wihout allowing the compiler to assume it doesn't happen.
Would C have been better if the spec was different?
Even something like register allocation requires knowledge of what pointers point to.
It would be more helpful for the compiler to assume that it does not know what will happen. This is how C actually worked for many years.
> a C program executing undefined behaviour is _invalid C_.
That was not formerly the case, and it is not always helpful to redefine C in this way. Sometimes you really are not trying to write portable code, and you really do want the behavior you know that the target machine will give you, even if the C spec doesn't require it.
If it is really necessary to generate random code when some anomalous situation is encountered, that should be a special option to enable dangerous non-deterministic if-you-made-a-mistake-we-will-delete-parts-of-your-program type behavior. I wouldn't consider that optimization though, more like disabling all your compiler's safety features.
People complain about dead code elimination all the time when we have these discussions.
Inlining break code that try to read the return address off the stack frame or that make assumptions about stack layout.
Loop unrolling might change the order of stores and load, which is visible behaviour if any of those traps.
I assure you that for each optimization, no matter how trivial, it will break someone code
If we don't know what will happen that is Undefined Behaviour.
The contradiction you have within yourself is that you know what you want to happen, but that's not what the specification says. If you want specific behaviour you need to specify what it is - not mumble and make a vague wave of the hand about "behavior you know that the target machine will give you" when you've no promise of any such thing. That would come at a cost, and of course you don't want to pay that cost, but that means you can't have what it buys.
Implementation-defined behaviour is a thing. Not knowing what will happen is not an accurate description of undefined behaviour. What the compiler does is assume that undefined behaviour doesn’t happen. When it does happen, it results in a contradiction, and logically every sentence is a consequence of a contradiction (see e.g. “Bertrand Russell is the pope”). That produces all those infamous bugs. Because just like every sentence is a consequence of a contradiction, every program state can be a result of UB. This is untenable.
That is an incorrect assumption, as it clearly does.
It is also incorrect given the standard text.
That’s my entire point. Compiler is free to make incorrect assumptions.
> It is also incorrect given the standard text.
According to C11 standard, section 3.4.3, the standard imposes no requirements on undefined behaviour.
In fact, in the rationale of the original spec., I remember reading that the C standard was expressly designed to be a minimal spec, and that just being compliant with the spec was insufficient for the resulting compiler to be fit for purpose.
And of course the original spec did specify a range of acceptable behaviors, and that language is, in fact, still in the standard. It was just made non-binding. However, it is still there, and pretending it is not seems disingenuous at best.
I agree, but that includes GCC and Clang. ¯\_(ツ)_/¯
And even more critical: Signed integer overflow should be implementation defined, and each implementation do something sane (different from assuming it doesn't happen). This would have saved us many security vulnerabilities, and unnecessary program crashes.
Can't you get that behaviour with -O0 or similar?
Beyond that, you've got a contradiction in your statement -- you can't "force correct behaviour" from a compiler at any point. The compiler always tries to generate correct behaviour according to what the code actually says. If you lie to the compiler, it'll try its best to believe you.
C compilers are intended to accept every correct C program. But they can only do this by also accepting a wide range of incorrect C programs -- if we can prove that the program can't be correct then we can reject it, otherwise we have to trust the programmer. Contrast this with Rust, where the intent is to reject every incorrect Rust program. Again, not every program can be clearly judged correct or incorrect, but in this case we'll err on the side of not trusting the programmer. Of course, "unsafe" in Rust and various warnings that can be enabled in C mean you can tell the Rust compiler to trust the programmer and tell the C compiler to disallow a subset of possibly-correct but unprovable programs, but the general intent still stands.
So if you want to write in a language that's like C but with "correct behaviour" then ultimately you'll have to procure yourself a compiler to do that. Because the authors of the various C compilers try very hard to have correct behaviour, and just because you want to be able to get away with lying to their compilers doesn't magically make them wrong.
The reasoning is something like "if we assume UB doesn't happen, but it does happen, the resulting behavior is unpredictable. This is allowed by the standard, though, because UB allows for any behavior, including that produced by assuming UB doesn't happen."
In other words, major implementations treat UB as preconditions. Violating those preconditions gets you Interesting Results (TM), but that's allowed by the standard because "unpredictable results " really means unpredictable results.
For example, null pointer dereference is UB. If an implementation assumes null pointers can never be dereferenced, it can better optimize some code. If it turns out a null pointer is dereferenced, the argument is that whatever happens then is still permitted by the standard as the standard does not define any program semantics for programs containing UB.
undefined behavior
behavior, upon use of a nonportable or erroneous program construct or of erroneous data,
for which this International Standard imposes no requirements
NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable
results, to behaving during translation or program execution in a documented manner characteristic of the
environment (with or without the issuance of a diagnostic message), to terminating a translation or
execution (with the issuance of a diagnostic message).
EXAMPLE An example of undefined behavior is the behavior on integer overflow
The important wording here is "this International Standard imposes no requirements"
i.e. an implementation is allowed to do literally anything in the case of undefined behaviour. It's not quite that the compiler writers are saying "this can never happen", it's more along the lines of "if this does happen, we can do anything at all, including acting as if the conditions were such that UB could not have happened."
So if you multiply two signed ints that the compiler knows are positive, the compiler can assume that the result can't overflow. Because, if it does overflow, the compiler can emit code that does absolutely anything in that case - including acting as if it didn't overflow. Therefore, it can elide checks for a negative result, because either there was no overflow in which case the check is redundant, or there was but the code is allowed to do anything at all - including not performing the check for a negative result.
I'm not sure if I agree with your interpretation of the spec, but even if that's the technically correct interpretation, arguing that things went wrong because the compiler miscompiled the program and that it didn't do the things it was supposed to before it was allowed to do literally anything... just isn't an interesting argument. Things went wrong because your program was wrong.
You might want a language that restricts this, but C is not that language. Or you might want a language that defines all its behaviours, but C is not that language either. Compiler writers put a lot of effort into making their compilers do exactly what the programmer tells them to do.
My personal take is that the correct response to the difficulties of ensuring your C program doesn't exhibit undefined behaviour is probably to avoid writing new code in C. But if you do still need to write C for whatever reason (which I do, occasionally) then it's only sensible to take as much care as the language design expects programmers to take: the compiler trusts the programmer to only attempt operations with defined results.
Honestly, I think we'd all be better served by pushing the concept of "undefined behaviour" a bit further into the background. C has defined behaviours, and the standard helpfully makes explicit which behaviours fall outside the definitions. If you want a defined output then your program had better have a defined behaviour when presented with your input.
I'm not suggesting this is ideal -- far from it, I avoid writing new C code. But it's what C does. If you want to avoid needing to make sure that your program only attempts defined operations, switch to a language that doesn't impose that requirement.
No it's not. Or let me rephrase that. The standard says the following:
Permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
Which of these is "the compiler is allowed to assume it does not happen"?
I agree with where you're coming from, but "ignoring the situation completely with unpredictable results" sounds like it pretty much fits the bill. Pretending something doesn't happen sounds a lot like ignoring it completely to me.
What the standard obviously does exclude is nonsense like intentionally reformatting your drives.
Quite the opposite. Assuming it doesn't happen (not "pretending"), is very much not ignoring the situation, at least if you then act on that assumption that it does not happen.
Ignoring it just lets it happen when it does, so if the program specifies an out-of-bounds access, the compiler generates code for an out-of-bounds access, ignoring the fact that it is an out of bounds access.
I'm not sure how assuming UB doesn't happen is distinguishable from a choosing to ignore the situation every time one comes up. You get the same result either way.
For example, a compiler can assume that null pointers are never dereferenced, or every time a null pointer is/may be dereferenced it can just "ignore the situation" with the dereference. I'm not seeing a functional difference here.
This is, of course, subject to the minor problem that "situation" is arguably underspecified. Compiler writers appear to interpret it as something akin to "code path" (so "ignoring the situation" means "ignoring code paths invoking UB"), while UB-goes-too-far proponents appear to interpret it more broadly, more like "the fact that UB can/will happen" (so "ignoring the situation" means "ignore the fact UB will/may happen").
> Ignoring it just lets it happen when it does, so if the program specifies an out-of-bounds access, the compiler generates code for an out-of-bounds access, ignoring the fact that it is an out of bounds access.
Why wouldn't this fall under "behaving during translation or program execution in a documented manner characteristic of the environment" instead?
strcpy(P, filename);
free(P);
if (P[0] == '.') {
// hidden file
// do something
}
There are good arguments for both sides, in my opinion. But let's not pretend they're the same thing. Deleting logic because it provably would result in UB is not the same as ignoring the UB.
1. This code path is executed. UB will be invoked.
2. This code path is not executed. No UB occurs.
If the compiler "ignores the situation" with UB, code path 1 is ignored (i.e., dropped from consideration). This probably results in the removal of the snippet as dead code.
If the compiler assumes no UB occurs, code path 1 is eliminated as cannot-happen, and the compiler probably deletes the snippet as dead code.
Same result either way (with the obvious caveat that this is one possible interpretation of "ignoring the situation").
> ignoring the situation [of undefined behavior] probably still results in doing something for hidden files.
The problem is that this assumes a very specific definition of "ignoring the situation" which, while understandable, isn't the only interpretation permitted by the Standard.
In addition, there's the fact that such an interpretation would arguably fall under "behaving during translation or program execution in a documented manner characteristic of the environment" instead.
> Deleting logic because it provably would result in UB is not the same as ignoring the UB.
True, but "ignoring the UB" isn't what the Standard says. It says "ignoring the situation", and that's the problem - people can't agree on what "ignoring the situation" is supposed to mean. Compiler writers appear to take it to mean "ignore UB-invoking code paths", UB-goes-too-far proponents take it to mean "ignore the presence of UB".
Ignoring something != killing something. There is no dialect of English in which ignoring something is compatible with eliminating it from existence.
EDIT: Note: I am not personally saying compiler writers are wrong. The body text says it imposes no requirements, so at the very least it's a reasonable interpretation to say the footnote text isn't binding and/or that "possible" behaviors are examples, not a full enumeration. But! On the narrow question of "ignoring" the behavior altogether... Hunting for UB via static analysis and then changing your output based on whether you find it simply is not what the word "ignoring" means.
What you are describing is ignoring the code that has the undefined behaviour due to it having undefined behaviour.
That is not ignoring the undefined behaviour, it is the opposite.
Again, it all comes down to interpreting "the situation". Compiler writers construe it broadly; "Ignore the presence of UB" (i.e., construing "the situation" narrowly) is another possible interpretation, but I don't think it's the one and only definitive one.
In addition, why isn't "ignore the presence of UB" covered by "behaving during translation or program execution in a documented manner characteristic of the environment"? See a null pointer dereference? Just do the "characteristic thing" during translation and emit the dereference. Maybe implementations will need to add documentation somewhere, but that's not exactly the challenging part.
What you are confusing is "being agnostic about something happening or not happening" and "assuming it cannot happen".
And sorry, the "situation" is pretty precisely scoped by "Permissible undefined behavior...". So what can be ignored is this instance of UB, not the fact that UB exists.
Otherwise, if you're going to arbitrarily expand the scope of what the situation is, then how about "the fact that a C spec exist?". That's a situation, after all, and it is the situation you are in.
Or maybe just ignore parts of the spec, like the ones that define what is UB and what is not UB.
Then everything becomes a trigger, and if I can expand scope like that, then I have a standards-compliant C-compiler for you:
int main() { int a = *-1; }
I doubt anyone would accept this broadening of the scope.
Once again, ignoring something is not generally the same as assuming it doesn't exist. They could be the same if you assume it doesn't exist and then do nothing differently. However, if you use your assumption that it doesn't exist and act differently based on that assumption than if it did exist, then you are not ignoring the situation.
And the latter is clearly what is happening with today's optimising C compilers. They act very differently in the presence of UB than they would if the UB would not be there, for example not translating code that they would have translated had the UB not been there, had it not been UB or had they actually ignored the UB as the should have.
> just do the "characteristic thing" during translation and emit the [null pointer] dereference
These things overlap slightly, but I doubt that "just emitting the dereference" qualifies as a documented exception to normal processing of a pointer dereference due to UB. It is exactly the same thing it does when the pointer dereference is UB, so it is just ignoring the UB.
Another misinterpretation that seems to be common is to interpret "the environment" in "characteristic of the environment" to include the (optimising) compiler itself.
But it is ignoring the situation? At least, given the broader interpretation of "the situation". I understand there's a narrow interpretation as well.
> And sorry, the "situation" is pretty precisely scoped by "Permissible undefined behavior...".
Maybe? I think I understand the argument. Will need to think on it some more...
> So what can be ignored is this instance of UB, not the fact that UB exists.
I think I haven't been clear enough on this - I had been using "ignore the fact that UB exists" to essentially mean "ignore this instance of UB" - i.e., carry on as if there was no UB. I had been using "ignore code paths with UB" for the broader modern-compiler-style interpretation.
> Otherwise, if you're going to arbitrarily expand the scope of what the situation is, then how about "the fact that a C spec exist?". That's a situation, after all, and it is the situation you are in.
Sure, it's a situation, but I don't think anyone is exactly advocating for an arbitrary expansion of the scope of a situation. "The fact that a C spec exists" is a situation, but it doesn't even pretend to have anything to do with the Standard's permissible UB.
> Once again, ignoring something is not generally the same as assuming it doesn't exist. They could be the same if you assume it doesn't exist and then do nothing differently. However, if you use your assumption that it doesn't exist and act differently based on that assumption than if it did exist, then you are not ignoring the situation.
I'd agree that proceeding without considering UB at all would count as ignoring something.
However, I'd argue that that's not the only way to read "ignore" - dropping something from consideration, to me, certainly sounds like ignoring something. You had to choose to do so, but that doesn't make it not ignoring it. That also depends on framing, though - back to how broadly "the situation" should be read.
> I doubt that "just emitting the dereference" qualifies as a documented exception to normal processing of a pointer dereference due to UB. It is exactly the same thing it does when the pointer dereference is UB, so it is just ignoring the UB.
Sorry, I don't quite understand what you're trying to say with the first sentence - where did the concept of an exception to normal processing come from? The idea was that emitting a dereference is the characteristic translation behavior, so that phrase in the Standard would cover "ignoring the UB" and doing what may otherwise be expected.
> Another misinterpretation that seems to be common is to interpret "the environment" in "characteristic of the environment" to include the (optimising) compiler itself.
I had interpreted "the environment" as including semantics; i.e., the translation environment includes these rules for translation/program semantics, so a characteristic behavior could be "normal" semantics. This interpretation doesn't need to include the compiler since the characteristic behavior is derived from the environment, not the compiler.
Looking more closely at the Standard, though, I'm not too confident in this interpretation. Perhaps "behaving during [] program execution in a documented manner characteristic of the environment" could work, though it's admittedly not what I originally had in mind, and I'm still not sure it works.
----
I do have to admit, though, that after the discussions I've had with you I'm less confident about my understanding of this. It'd be nice to talk to an actual major compiler dev or some C89 committee members about this. Feel like I had run across such a thing at some point, but I don't remember where or when.
> Hunting for UB via static analysis and then changing your output based on whether you find it simply is not what the word "ignoring" means.
Why not? A static analysis pass can flag a code path as containing UB, and future passes can then ignore that path. Sure sounds like "ignoring" to me.
It is fundamentally impossible to both ignore X and make decisions based on X at the same time.
Something cannot be both ignored and a key criterion.
Your argument is akin to saying that a company hiring process that "ignores race" means eliminating applicants based on their race. It is untrue. It is the opposite of true. And I think you know that, so please stop trolling.
Of course you can - the former can be the action you take as a result of the decision. Choosing to not consider something is distinct from refusing to make a decision based on that something, but both are "ignoring" that something.
Again, this boils down to how "ignoring the situation" is interpreted. I can ignore situations with UB and proceed as if those situations aren't present, or I can ignore situations with UB and proceed as if the UB weren't present. The Standard's wording does not rule out one or the other.
In addition, why would the "ignore the existence of UB" not fall under "behaving during translation or program execution in a documented manner characteristic of the environment"? That seems to match "pretend the UB were not present" much more closely.
> Your argument is akin to saying that a company hiring process that "ignores race" means eliminating applicants based on their race.
No, it means that the hiring process makes decisions without considering what race-based effects that may have. If that happens to result in weird race-based outcomes, then that's what happens.
What is doing, in the extreme case, is ignoring it and not generating asm statements for it. Then again, how it could? Code that will trigger UB has no meaning so the compiler wouldn't know what code to generate.
Of course a compiler could assign meaning to some instance to UB.
For example I'm pretty sure that in GCC dereferencing a null pointer is not UB, but it is expected to trap (because POSIX) and the execution not to continue except via abnormal edges (exceptions or longjmp). This means that any code that can be proven to be reachable only through a nullpter dereference is effectively dead code, so in practice it can still introduce bugs if it didn't trap.
The normative text that specifies the semantics of undefined behavior is:
behavior [...] for which this International Standard imposes no requirements.
2. It used to be normative.
That interpretation is the root of the problem. Compilers authors use it to implement outright user hostile behavior in the name of elusive performance.
Why would you spend resources looking for zero days when you can have a few LLVM contributors plant them in every program "as an optimization"?
You can use the same compiler with a language whose design committee isn't deliberately user hostile, like Rust (where UB-like behaviors in safe code are considered soundness bugs).
Whether things should be UB or implementation defined requires a good value judgement. Per se there is not a real need of UB, (assuming that the behaviour of a machine can be specified), but on the other hand there isn't much value in specifying a situation where all control is lost to the point where implementation details leak (runtime structures are overwritten etc.).
Since there is (I assume?) an expectation for "implementation-defined" to have an actual definition, pedantically an implementation leak would require all implementation details to be codified, and thus set in stone.
By making things undefined we can avoid a large amount of complex code to detect a situation that shouldn't happen.
I recall IBM’s AIX used to put a zeroed-out block at address zero, so this code was guaranteed to always return 0:
*(int *)NULL
It’s not up to the platform. If it looks like a code path might dereference a null pointer, the compiler can and will make wild and bizarre optimizations in the surrounding code.
Yes, some compilers will make "wild and bizarre" optimisations (they are neither) based on the assumption that any pointer which is dereferenced is not NULL - because if it were, the code could have done the exact same thing anyway, because there are no constraints in that case.
But compilers don't have to do that. There are no constraints. A compiler can emit code to do anything at all in the event of a NULL pointer being dereferenced, including emitting the machine code you'd naively expect - which may trap on some CPUs, or always return 0 on reads and swallow writes on other CPUs. It may also emit code that wipes your hard disk (and still counts as a conforming implementation). Or make demons fly out of your nose.
A compiler vendor may choose to guarantee a specific behaviour for some invalid constructs. They are allowed to do this because there are no constraints on what they can do. It's just that most don't, because 99% of people comparing compilers care more about benchmarks than they do about what happens with invalid code.
So in some ways, it is up to the compiler/platform/CPU what actually happens with code that has invalid behaviour. But as a C author, you can't assume any particular behaviour, because it might change from one CPU to another. Or from one OS to another. Or from one compiler to another. Or from one version of one compiler to another.
It basically means that the whole portability argument that is supposed to be in favor of C is just wrong because every compiler and platform actually brings its own C dialect and it is just sheer coincidence that it works at all.
Not sure about the C code bases you've been spending time in, but in the ones I've looked at, the overwhelming majority of the code has been well-defined by either the C standard or the implementation. In the cases where it hasn't, the vast majority of those cases were genuine bugs that needed fixing, and the fix made the code well-defined.
The number of codebases I've seen that actually relied on unspecified behaviour, or on whatever the current compiler/OS/CPU happened to do with undefined behaviour, is miniscule. (Or an entry in an obfuscated/underhanded coding contest.)
Since a compiler cannot predict in all cases in advance whether a pointer is invalid it should be required to compile a pointer dereference the same way in all cases, i.e. emit the same code regardless of whether it has determined the pointer is invalid or not. Otherwise it effectively emits random code depending on how the optimizer is feeling.
Non-deterministic behavior for the same operation on the same values in the same program is not very helpful behavior, whenever and to the degree it can be avoided.