Type Punning Functions in C
evanmiller.org
evanmiller.org
Or plain abstraction violation. “Black magic” gives it an undeserved good reputation: “I must aspire to be clever enough to understand and actively make use of this.”
In fact, so far as I can tell by looking at the first example, the compiler can circumvent this whole thing by noting that:
1. DoubleToTheInt is only ever called in the same unit, so it can be inlined - no need to generate the function.
2. Pointer to DoubleToTheInt is created, but is then immediately cast to an incompatible function type, which is undefined behavior - and so it can be substituted with any random value, say, whatever happens to be in EAX at that moment (any U.B. is substitutable for any other U.B.!), and the compiler can still avoid generating the actual function.
From http://catb.org/jargon/html/B/black-magic.html
> black magic: n.
> [common] A technique that works, though nobody really understands why. More obscure than voodoo programming, which may be done by cookbook. Compare also black art, deep magic, and magic number (sense 2).
Incidentally, if anyone is thinking of trying this for yourself: always remember to mlock() the memory containing your summoning. The last thing you want is some demon of the pit getting swapped out to disk. Not only is swap notoriously insecure, but sulphur buildup on the disk platter will cause a disk crash. And for heaven's sake, make sure your pentacle is properly aligned. If it's split across two pages, they may end up not being next to each other in physical ram, and then it won't work!
Ideally, use an embedded device where you know what every cycle does. Don't be put off by the low power. You'd be surprised how much evil you can do with an Arduino.
https://github.com/facebook/buck/commit/48f1e201a966b9e2adab...
The result is the same though; I forgot that signal mask is not reset on exec. Damn POSIX signals!
If you write this in production code, and don't get sent home that day in tears, your company's code review process has failed.
gcc 4.8.4 on Ubuntu 14.04 Trusty still gives no warnings, but gives a different result:
$ gcc -W -Wall x.c -lm
$ a.out
(0.99)^100: 0.366032
(0.99)^100: inf
$ clang -W -Wall -fsanitize=undefined x.c -lm
$ a.out
(0.99)^100: 0.366032
(0.99)^100: inf
; gcc -v |[2] grep version
gcc version 6.1.1 20160802 (GCC)
; gcc -Wall -Werror ./something.c -lm
; ./a.out
(0.99)^100: 0.366032
(0.99)^100: 0.000000
;Another thing which makes this sort of thing difficult to warn in is the brokenness of autoconf. Autoconf checks that your system has memcpy (or anything that you check via AC_TRY_LINK) by compiling this program:
void memcpy();
int main() {
(void)memcpy();
}However, the snippet above is still broken in that respect, because it doesn't match the return type of memcpy - and that is U.B.
Yes. And the call here does not match the types of the real parameters, even before we consider the return type.
> I guess they could have wrapped it in if(0){...} to make this completely unambiguous.
Depending on optimizations, that may mean the executable does not depend on the symbol we're trying to test the presence of.
Indeed, on my local gcc that's the case, even with -O0.
$ cat test.c
void *memcpy();
int main() {
if(TEST) {
(void)memcpy();
}
}
$ gcc -DTEST=0 test.c
$ objdump -T a.out
a.out: file format elf64-x86-64
DYNAMIC SYMBOL TABLE:
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __libc_start_main
0000000000000000 w D *UND* 0000000000000000 __gmon_start__
$ gcc -DTEST=1 test.c
$ objdump -T a.out
a.out: file format elf64-x86-64
DYNAMIC SYMBOL TABLE:
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __libc_start_main
0000000000000000 w D *UND* 0000000000000000 __gmon_start__
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.14 memcpy
$ gcc -O0 -DTEST=0 test.c
$ objdump -T a.out
a.out: file format elf64-x86-64
DYNAMIC SYMBOL TABLE:
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __libc_start_main
0000000000000000 w D *UND* 0000000000000000 __gmon_start__
$ cat test.c
void *blah();
int main() {
if(TEST) {
(void)blah();
}
}
$ gcc -DTEST=1 test.c
/tmp/cc2Vk5mE.o: In function `main':
test.c:(.text+0xa): undefined reference to `blah'
collect2: error: ld returned 1 exit status
$ gcc -DTEST=0 test.c
$ objdump -T a.out
a.out: file format elf64-x86-64
DYNAMIC SYMBOL TABLE:
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __libc_start_main
0000000000000000 w D *UND* 0000000000000000 __gmon_start__
$ gcc -O0 -DTEST=0 test.c
$ objdump -T a.out
a.out: file format elf64-x86-64
DYNAMIC SYMBOL TABLE:
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 __libc_start_main
0000000000000000 w D *UND* 0000000000000000 __gmon_start__You definitely have a point regarding compiler just ripping that code out. I guess the proper approach would be to do something like if(argv[0][0]), or test a volatile variable.
Yeah? So which compiler do you have in mind exactly we should use?
The example code uses C style casts, which is C-speak for "fuck off compiler I know what I'm doing." I'd like to see what warnings your compiler produces. VS2015 /W4 /ANALYZE produces nothing, and neither do clang nor GCC from the looks of it.
That people C cast without knowing what they're doing is rather unfortunate tragedy. That this is a common enough problem, that the Win32 API is littered with backwards compatability hacks where someone C-casted away ABI differences turns it into a statistic.
> If you write this in production code, and don't get sent home that day in tears, your company's code review process has failed.
Assuming we're talking "well meaning newbie" instead of "sanity hating self-flagellating freak and/or undefined behavior worshiping cultist", I prefer to carefully explain that they've invoked undefined behavior, that undefined behavior leads to crunching to find optimizer induced heisenbugs in the weeks before we ship, and to impress upon them that this is why we never cast away even "trivial" and "minor" function pointer differences. And then, when they ask what the correct thing to do is, point out the API docs and header both have this little "WINAPI" annotation they can use. Hey! No more casts! Easier than the casts too. Win/win.
However, if we are talking about the "sanity hating self-flagellating freak and/or undefined behavior worshiping cultist", I can only recommend nuking the site from orbit. It's the only way to be sure. If their home isn't part of the radioactive crater, you didn't use large enough nukes, and your company's code review process has failed.
If you need something like that, just drop down to assembly. It's more portable (in the sense that you're coding to an interface), infinitely more clear about what you're doing and and how you're doing it, much less likely to break, and when it does break is likely to break loudly.
Even if you've never done a lick of assembly programming (I haven't done much) it'd still be infinitely preferable, even if you had to support a dozen architectures with two dozen different function proxies.
"Hack" is a very broad term. By some definitions a significant chunk of extant code qualifies as a hack. But whatever the definition, a hack still implies some quantum of legitimacy. There's nothing legitimate in using function type punning as a way to reorder or control argument values. Not unless this was 1969, where there was precisely one C compiler, used by precisely one guy, and where such code was expected to be thrown out in short order, never to infect code that executed in other than identical circumstances.
double IntPowerOfDouble(int power, double base)
{
return DoubleToTheInt(base, power);
}
I used to believe that there were times when dirty tricks were the right call in certain cases. The problem was that the long term view always proved me wrong: a more verbose but commonly understood approach is much more maintainable than a concise dirty trick, even with ample commenting and documentation.
Yes.
However.
If you wish to convince me that any of the type punning examples from the article, as given, are useful in production code, I will allow you that opportunity. But only after I've had the chance to make myself a bowl of popcorn large enough to share with whoever else might be entranced by the ensuing trainwreck.
For example, having to explain the functional use of 'x = !!variable' to a C programmer. There are many like that.
Sometime you just have to decide to raise the bar a little...
I often remind my fellow engineers to keep their code muggle-friendly. And, mind you, our code base is mostly Python, a language that actively avoids black magic by design.
What a great phrase, I'm going to borrow that.
And sometimes you need to do terrifying, nonportable, ugly hacks in order to get something specific to work. My own contribution to this field is an ARM stack unwinder that requires only one __asm__ inline in order to get the link register and does everything else by interpreting instructions.
Undefined behavior in C sucks, but being its presence or absence being undecidable is not the main reason.
It's only undecidable if the language makes it undecidable. For example, subject reduction is decidable in a type-safe language (EDIT: with decidable type checking, oops!), because the very definition of type safety means that typability implies subject reduction.
Of course, not everything of interest can be made decidable in a general-purpose language, but the integrity of a language's basic abstractions should never be in question.
Any interesting property is undecidable in the sense that we will be unable to produce an answer for some programs. Nothing says that there can't exist plenty of nontrivial programs for which we can show the property holds.
While almost certainly the case, I'm not sure how we can demonstrate that this is true in principle without begging the question, if we're asking whether human intelligence is computable.
I agree that all we have evidence of is that we can solve specific cases, and that this completely undermines the argument you're critiquing.
This error also occurs more subtly. I had someone tell me that static type checking was basically useless because it couldn't even do something as simple as saying whether a program will halt. Nevermind, if we're demanding this of all programs, that neither can any other method of reasoning we're applying. (And also nevermind, if we're demanding this of an interesting subset of programs, that there are type systems that will tell us exactly that...).
I cheated a little bit above — I assumed you're running code on a 64-bit x86 PC. If you're on another architecture, the trick above might not work. In spite of C's reputation for having an infinite number of dark corners, the int-double-argument-order behavior is certainly not a part of the C standard. It's a result of how functions are called on today's x86 machines, and can be used for some neat programming tricks.
There are varieties of type punning that are legal C, so I agree it's confusing to present this UB implementation artifact as "Type Punning ... in C".
Still a neat magic trick, and an interesting explanation of how it works.
Your response would make sense if twoodfin claimed it was implicitly undefined behavior. But that's not the case. The standard clearly addresses this case.
If a "shall" or "shall not" requirement that appears outside
of a constraint or runtime constraint is violated, the behavior
is undefined. Undefined behavior is otherwise indicated in this
International Standard by the words "undefined behavior" or by
the omission of any explicit definition of behavior. There is
no difference in emphasis among these three; they all describe
"behavior that is undefined".
i think in your cosmic ray example, you get well defined behaviour producing unexpected results - its a subtle distinction, but that's part of why UB is horrible, its results can be even less reasonable than those of random bit twiddling.
e.g. your standard conforming addition function will still always perform addition - now if by definition that function merely looked like an addition but was actually describing UB, the result could be anything at all, including multiplication (unlikely, but possible) and doing nothing at all (very likely).
Undefined behavior is undefined behavior FOR C. Undefined behavior makes no sense unless referring to a specific language.
Besides, people have been taking advantage of certain undefined behaviors in actual C programs since the dawn of time -- deliberately I mean.
For example, POSIX now mandates that the NULL pointer has all zero bits: http://austingroupbugs.net/view.php?id=940
Representations are readily observable because the standard permits all objects to be cast to char type. Therefore the standard basically says that you can read and even modify objects through a char array as long as you obey house rules.
So POSIX isn't changing what undefined means by mandating that CHAR_BIT is 8, or that null pointers can be represented as all-zero bits.
By contrast, I think the definition of dlsym() does depend on undefined behavior because C does not permit casting function pointers to void pointers or vice-versa. For various reasons that's a significant deficiency. Keeping that behavior undefined is intended to, among other things, preserve the ability to implement tagged pointers in a standards compliant manner, or to even create implementations for Harvard architecture machines. Also, I think the distinction between undefined behavior and unspecified or implementation-defined behavior preserves the ability to implement practical, quality static analyzers that can analyze all C code, whether or not dependent on certain implementation-specific behaviors. Some amount of license is a practical necessity; too much makes the standard useless.
C permits assignment of pointers to functions of incompatible types as long as the functions aren't invoked. But mixing pointers to functions and pointers to objects is a definite no-no. So POSIX will probably add a sister to dlsym that returns a generic function pointer type like (void (*)(void)).
I think that change would be a mistake: it makes no practical difference on the machines for which POSIX is intended, so this sister function would bloat the API only to satisfy pedantic concerns that don't really matter. Besides, POSIX systems have to keep the traditional dlsym around for compatibility, which further erodes the advantage of your proposed function.
Right now, POSIX mandates by implication that interconversation of function and data pointers works, and I think that's a perfectly reasonable state of affairs. There is nothing wrong with POSIX constraining the C standard.
For example, C++ class vtables are a prime target for heap overflow attacks because they store function pointers on the heap. And with the anti-ROP measures coming around the corner it's a target that will be increasingly used to create and execute a ROP chain.
So, for example, I can easily imagine new compiler modes and coding styles that outright ban casting function pointers to object pointers, and ban casting pointers to function pointers to other object pointers. PaX and W^X don't help you here as the issue isn't writing over code, but writing over pointers to code.
Banning those conversions is neither sufficient nor necessary, but nonetheless a very plausible and practical counter-measure. In that context dlsym sticks out like a sore thumb. One of the easiest solutions is to just standardize a routine for returning a function pointer. That's arguably better than forcing compilers to understand the semantics of dlsym.
Note that implementations can still support mixing pointer types, and POSIX could even still require it so existing software can continue using dlsym the same way. But it would simplify things for the tooling and runtimes that enforce the distinction. Given that it makes POSIX more consistent with the C standard, it's a win-win.
Is that so? The actual pointers should be in .data. The pointer to the vtable is on the heap (or on the stack, or wherever your object lives). In any case, you should be able to use something like [1] to place constraints on indirect calls, right? There are lots of approaches to double checking function calls that don't involve increasing function pointer size.
Besides, there are lots of bits of code that depend on function- and data-pointer interconvertibility --- people can stuff function pointers in raw void* cookies to callbacks.
Rather than make the effort to make existing code work in this (in practice, new) subset of customary (as opposed to standard) C, wouldn't it be better to just write components in a language that's memory-safe to start with?
[1] https://gcc.gnu.org/wiki/cauldron2012?action=AttachFile&do=g...
Restricting void conversions for function pointers is just a pragmatic example where part of the burden of implementing and enforcing new language features can be paid by reference to the C standard and the license it affords. It's loosely related to tagged pointers in that both depend on software not abusing the type system too thoroughly. And as problematic as C's type system is, it's not without rhyme or reason.
We don't _need_ to rely on the the C standard in any way, shape, or form to implement counter measures. But the function of a language standard is to be a locus for these things, and to help ease the burden of coordinating and implementing changes, and pushing adoption among a wide and diverse community.
As for simply just switching to some other language, I don't think that's even remotely realistic. C isn't going away anytime soon, and neither are people going to stop writing new C code anytime soon.
People rave about a language like Rust, but there is no language called "Rust". It's constantly being updated, claimed guarantees regarding compatibility notwithstanding. You simply cannot rely on Rust to create a large and complex ecosystem of software yet, and probably won't for awhile. Conversely, when authors of a language like Go adamantly refuse to evolve the language out of legitimate concern for compatibility, people flip out. Language maintainers are damned if they do, damned if they don't. There's no simple answer or approach.
C has survived for as long as it has partly _because_ undefined and unspecified behavior provide space for it to slowly evolve through controlled experimentation. And it will continue doing so (and should do so) on its descent just as it did on its ascent.
Updated does not mean incompatible. And the community bears this out, in the survey, most people had things Just Work, and for those that had issues, they said the upgrade was incredibly minor.
If there's something different than this, I'd love to hear about it: we care deeply about stability.
1) The issue isn't just modifying vtables; again, that's just a classic example intended to highlight the issue. Both C and C++ code can and often does juggle pointers to regular functions.
2) The referenced mechanism to protect vtables and calls through vtables _rely_ on the much stricter language semantics regarding method invocations. My entire point was that by enforcing some of the restrictions C already guarantees regarding function pointers, we can more easily implement similar approaches for C and C++ code directly manipulating function pointers.
Your example about void pointer cookies for callbacks is perfect. Restricting conversions between function and object pointers makes it easier to add instrumentation to verify integrity. I'm sure it's possible to do that without such limitations, especially with your particular example, but it adds complexity and removes opportunities for optimizing away the cost. Undefined behavior helps compilers improve performance because it simplifies the model compilers require for proving the correctness of generated code. That same dynamic also makes it easier to implement and enforce security constraints. That simpler model can allow the compiler to assume a pointer is valid, but it also permits it to more easily detect and abort the program when it isn't without violating the contract with the programmer.
Contrast that with a language that permits you to dynamically update code sections by poking and jumping to memory anywhere. That permits all kinds of fun hacks, but it means the compiler can't enforce any meaningful constraints. There's a huge middle ground between that and a language like Rust or Haskell.
I find it odd that people will in the same breath claim that C is a security nightmare and should be abandoned, yet argue to preserve insecure semantics that could be easily removed and in many cases are already disallowed by the standard. That's a harmful cognitive dissonance. To be clear, I'm not accusing you of that; but I find your arguments to be uncomfortably close to that misguided conception of C and the C ecosystem.
I have known of organizations that are happy only when code compiles without warnings.
int i = 16 >> 32;The platform ABI is guaranteed to work in a certain way.
In this specific case, the compiler is not allowed to make any assumptions about the legality, but has to follow the ABI.
There's no UB involved (not every platform specific issue may be labeled UB; think accessing bare metal memory on one platform where a specific address is IO mapped and on another it is regular SRAM; it's perfectly valid C, and has perfectly known semantics on both platforms, but will work on one platform and not work on another platform; this is also not UB).
To be clear, UB means the compiler is allowed to e.g. totally remove the call (or really anything else it wants to do at all) if it can be positive that this is what's happening -- not that it will behave as could be expected on the platform, the term for that would be implementation defined behavior.
[0]: http://c0x.coding-guidelines.com/6.3.2.3.html number 766-768
/*VARARGS2*/
int margin_printf (fp, a, b, c, d, e, f, g)
FILE *fp;
char *a;
long b, c, d, e, f, g;
{
ind_printf (0, fp, a, b, c, d, e, f, g);
} /* margin_printf */
margin_printf (outfile, length ? "/* %s */\n" : "\n", storage);
K&R C actually has a minimalist elegance to it that later versions of C lost. Not that I'm claiming that it's better, of course. But K&R C had a distinct philosophy to it that was definitely its own.
I don't know about "occasionally". At least on Windows, calling conventions used to be a constant headache. The C language default was cdecl, but the Win32 API default (most of the time) was stdcall. Then there was a variant (fastcall?) that tried to make use of registers.
Maybe it's all fixed now... But this being Win32, I rather expect they've managed to accumulate a few more "interesting" gotchas and edge cases as the platform expanded with 64-bit, WinRT, Universal Windows Platform and whatever.
Not anymore: https://blogs.msdn.microsoft.com/vcblog/2013/07/11/introduci...
i would not call that an improvement in all cases... the extra code produced by enabling exceptions, even for the latest compilers, produces enough bloat and slow down for basically everyone seriously concerned with real-time performance to turn them off.
small simple functions are forced to have prologue/epilogue, use a frame pointer and to preserve the return address on the stack. doesn't sound like much, but if your function does anything simple (float f() { return 1; } bool g(float a, float b) { return a < b; } etc...) you spend all the time paying for exceptions instead of the intended functionality.
there is a table i remember in some doc about this, the only reason i listed from memory was the lack of being able to find it quickly on google
.. its a shame i can't find it. perhaps my memory is imply faulty.
the "constant headache" you allude to played no small part in that decision i would say...
unix/windows are not interchangeable with compilers targetting those platforms. the article doesn't make the distinction or highlight why it matters - the OS has nothing to do with calling conventions in this context, they are just very conveniently the same in this case.
this example doesn't work with vs2010 out of the box because it targets win32 by default... for example. its the compilers, and only the compilers, which are deciding this. the standard calling convention for the platform is incidental, and it being identical to the default for the compiler is a wise design choice for compilers targetting x64, rather than a necessity - something which is much clearer in the old world of windows and x86 (32-bit) where cdecl was the default out of the box, and stdcall was what windows api calls expected.
(At the ABI boundary, anyway. There's also __vectorcall now. And, of course, the compiler can use whatever it wants for functions that are not exposed at the boundary - even arbitrary register assignments based on where the function is actually called, to minimize the amount of work needed to save registers etc.)