Undefined Behavior in LLVM [pdf]
cs.utah.edu
cs.utah.edu
For the first group of commenters, please don't bother posting again, just search HN and you can find a lifetime's supply of C love/hate arguments.
For the second group of people, I'd recommend reading two fantastic guides to UB:
http://blog.regehr.org/archives/213
http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
These should be enough to explain why UB is trickier than you think...
Using another language, aside from c++, would involve using some kind of ffi mapping, made by me or someone else. So now I not only have to understand the semantics of c and the libraries I'm using I have to understand the semantics of the ffi mapping.
Using the ffmpeg libraries I've had to dig into the source to figure out what it's doing[0]. If I'm using python(or whatever) now I have to do that and figure out what the mapping is doing. And I'm still not safe from segfaults and memory leaks, they're just harder to find and understand.
Soft realtime is trivial in C. If I'm capturing a video stream at 10fps I have about 100ms to do everything to handle a frame. This is very easy to do in c. If I'm using a garbage collected language I have 100ms-X, where X is the longest garbage collection pause.
I tried doing a similar project in java and found it very difficult. Using any library, for example to output a jpeg, created tons of garbage, resulting in long gc pauses. I tried a bunch of different -XXasdf options that helped but to really solving the problem would involve reimplementing a bunch of stuff to reuse buffers and avoid garbage. I freely admit this may be because of my ignorance of how to do it correctly but again it's trivial to avoid these pauses in c.
So I ask sincerely what should I do? I'll gladly throw out the years of painfully acquired c knowledge if you have a better solution for me today.
[0] I wonder why avformat_find_stream_info is detecting the from framerate? Good luck :) http://ffmpeg.org/doxygen/trunk/libavformat_2utils_8c_source...
Even so, I would (and have) still use an FFI and put as much as possible in a safer language. You will get segfaults and memory leaks because you're still calling C code, but you won't be adding more, and IME that's well worth the overhead of an FFI. (Also ffmpeg has had quite a bit of attention from people using valgrind, afl-fuzz etc, so its defect rate is lower than your C code would likely have, unless you're also using those tools). Alternatively, depending on your use case you might be able to separate the decoding/encoding stages from the processing, in which case you can write the code without needing much in the way of libraries, and pipe to/from ffmpeg.
Write a quick prototype and see if you can get the performance you need within GC or a small amount of -XXasdf (you might be surprised, particularly with the state of Haskell and OCaml today. I've done literally this from Java, JNI-ing the ffmpeg libraries to do realtime streaming, and had it work fine, and that was five years ago). In the rare case something like that doesn't cut it Rust is what I'd use (there are other options but I know less about them).
For example, I understand why signed integer overflow and violating type rules are UB; the first allows the compiler to optimize many inequalities, while the second enables various optimizations that rely on alias analysis.
But I can't understand why e.g. use of an uninitialized variable or oversized shift amounts are UB. Unless these also enable some powerful optimizations (the article doesn't mention any, and I can't think of any either), it would be much better to leave them implementation defined, not undefined. In that case, reading an uninitialized variable could result in any value, but it would still make the whole program valid.
Edit: should have said "platform-defined" instead of "implementation-defined".
Still, crashing would be a valid "implementation-defined" behaviour in my book.
http://en.cppreference.com/w/cpp/types/numeric_limits/signal...
sNaN doesn't actually trap - it just records a special bit in a FPU exception register. So yes, it can affect the control flow - which is something you should expect reading aribtrary values!
https://blogs.msdn.microsoft.com/oldnewthing/20040119-00/?p=...
Now the definition could vary between doing something that seems obviously correct, crashing or even deleting every file on your disk but whatever it does is up to the compiler implementer.
So, basically, only the native code emitter would take these behaviors into account, and could do whatever (e.g. emulate, crash, emit instructions and hope for the best, ...). But the rest of the compiler pipeline (optimisations) would still need to treat the code as valid.
For undefined behavior, the standard makes no requirements at all on the compiler.
Probably some obscure architecture traps on oversized shifts or something similar. To correctly implement it you would first have to check if the shift lies inside the bounds with a branch. Branches are expensive - this is the powerful optimisation.
if ((1 << 33) == 0) {
// whatever
}
I'm fine with my program crashing. I'm not fine with my compiler simply assuming this cannot happen, and removing the whole `if` statement.What's the difference? Do you mean you don't want your compiler to ever reorder instructions? That would cost a lot of performance. Try to formalize the guarantee that you want the compiler to give in the presence of programs like that, and the reasons for leaving it undefined become a lot more clear.
The compiler's job is making code run faster while preserving intent. If I can write a 33bit shift in ASM, then I should be able to write it in C. How the CPU executes the code is my (the programmer's) business, not the compiler's.
Suppose a CPU, when shifting a 32 bit register, uses only the lower 5 bits of the shift count. Then shifting a register by 32 will be a shift by 0, i.e. a no-op (instead of clearing the register). The compiler would have to generate additional code to correctly handle this which would be slower. So it was decided to make oversized shifts UB.
Exactly the stupidity I was pointing out - why make it undefined behaviour, when you could make it implementation-defined behaviour. You might argue the effects are the same, but they aren't, of course - the first is global (the whole program is invalid), whereas the second is local (only that instruction has an unknown (but probably still deterministic) outcome, and only in specific circumstances).
Undef variables taking on different values at different use points can lead to e.g. a == a being false. Code paths being possible with undefined values, that wouldn't be possible with any value sounds like a terrible idea to me, possibly exploitable, at least a source for heisenbugs
When llvm finds undefined behaviour in a code path (not possible undefined behaviour, but there actually is undefined behaviour), it does strange things instead of just trapping. When the execution reaches that code path, the program already is in an erroneous state, no need to optimize stuff there.
For instance:
int main(int argc, char argv) { int a = 1; int b = 0; printf("%d", a/b); }
compiles to:
define i32 @main(i32 %argc, i8 nocapture readnone %argv) #0 { %1 = tail call i32 (i8, ...) @printf(i8* getelementptr inbounds ([3 x i8]* @.str, i64 0, i64 0), i32 undef) #2 ret i32 0 }
Why let the program behave buggy when llvm can see that undefined behaviour will occur? Why not just trap instead? That would have zero cost.
It's not quite that easy. Because compilers these days have a large number of phases/optimization passes it can be prohibitively difficult to trace back from the location of a potential UB to what might/does trigger it (in the source). So what compilers do is ASSUME that no UB can occur, hence there's no need to generate a check for "b=0".
You can run clang with -fsanitize=undefined, though, and get runtime-checks for many UBs. It'll cost you runtime performance, though.
Maybe then they'd understand the cost better.
Compilers should generate warnings on all compile time UB and maybe even do data flow analysis for possible runtime UB.
Although, IMHO, code with a reference to block scope uninitialized variable shouldn't even compile.
People who want to get every last bit of performance out of their systems do want to write code like:
int x;
...
if(condition1) x = expression1;
...
while(expression2)
{
x = DoSomethingWith(x);
}
where the intelligent reader knows things are fine, but a compiler has a hard time proving it. For example, a contract implicit in an API may enforce the invariant, but that cannot be deduced from the code.Funny story, that is actually what caused Debian's weak keys fiasco. The code was using uninitialised memory to seed the PRNG and someone removed it because of valgrind warnings. So sometimes UB is exactly what you want. :P
int main() {
int *p = (int*)malloc(sizeof(int));
int *q = (int*)realloc(p, sizeof(int));
*p = 1;
*q = 2;
if (p == q)
printf("%d %d\n", *p, *q);
}
$ clang -O realloc.c ; ./a.out
1 2
That's about as eyebrow-raising as you can get!But yeah aliasing that the compiler can't see (if it's not doing a stamp based on the equality condition) I can understand why that's undefined as what else would you want the compiler to do? Treat all memory as volatile? That would destroy performance.
According to the standard, realloc doesn't invalidate its pointer parameter, since it can return the existing pointer (if the resizing could be done in-place) and can not touch the existing pointer (if the reallocation fails).
The only case in which the old pointer is not invalidated is if realloc fails.
As far as I know, in practice, at least with GCC, but likely clang as well, the compiler doesn't tracks the invalidation, but it simply assumes that any non null pointer returned by realloc (or any allocation function in fact) does not alias any other existing pointer in the program.
http://pubs.opengroup.org/onlinepubs/7908799/xsh/realloc.htm...
You know, like Rust.
So do that then.
And dangling pointers result in use-after-free, which frequently results in remote code execution vulnerabilities. Do you have a practical alternative to eliminating those RCE vulnerabilities, or should we accept them?
To be clear, your position is an intellectually coherent one to take. But we should be clear about its implications.
- The reason why we use GC pointers in Servo is that JavaScript is garbage collected, and we can't do anything about that. If not for that, we probably wouldn't use GC at all.
- In fact, to prove this point, we don't use GC anywhere except the DOM. Most of my code is layout and rendering, which is GC-free, written in the safe language, and is very much not crippled in any way.
- GC pointers are designed to be safe. There are some known safety holes right now, mainly due to lack of time to plug them and wanting to balance the time spent to fix up the current system against the time needed to add proper GC support to Rust.
> The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object in the space allocated (until the space is explicitly freed or reallocated).
(Emphasis added.) After realloc() succeeds, you can no longer use the original pointer.
First argument to realloc takes a pointer of type "void star" and you passed in a pointer of type "int star" without casting it and this suggests you did the casts out of habit and i think its time to break the habit :-)
ps: had to write "void star" because the "star" character was not showing up for some reason.
...or shows someone more familiar with C++ than C. Implicit casts between pointer-to-void and other pointer types are not allowed in C++.
Here are some examples: http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
Anyone got any more? Please, anecdotes help explain this! :D