#define a stdio
#define header <a.h> // five tokens: <, a, ., h, > (otherwise preprocessor wouldn't substitue "a")
#include header // one token after substitution: <stdio.h>
int main() {
struct {
int h;
} stdio;
a.h = 0;
printf("%d\n", stdio.h);
}
What happened here is:1. In #include directives a thing like <stdio.h> is one token; everywhere else (including other preprocessor directives) it is five tokens;
2. But an #include directive can have multiple tokens on the right hand side of "#include" before macro substitution. After macro substitution the strings are lexed again - into ONE token.
3. In main() "a.h" is substituted with "stdio.h" - this time it's THREE tokens.
(Oh, and writing "#include <a.h>" will produce an error, because here we have a single token <a.h>, so no "a" to be substituted, and normally there is no "a.h" header file in include path.)
So the same string gets lexed into a different number of tokens in different places; and that happens even after it got lexed first into the same number of tokens before macro substitution.
Now have an example of a C file that compiles with GCC, but not with Clang:
#define CMPS x /*
*/ <y>
#define ALMOST_SHIFT<CMPS
#define FWD_FST(x, y) x
#define CONCAT(x, y) x ## y
#define x
# /*
LOL
*/ define y stdio.h
#include CMPS
#undef x
#undef y
#include FWD_FST(<stdint.h>, y)
// btw. comma is a valid character in a header name. but this will work anyway,
// because lexing in C works in mysterious ways
#include CONCAT(<std,def.h>)
int main() {
int x = 0;
int y = 0;
int z = 0;
printf("%d\n", CMPS z);
// printf("%d\n", z<ALMOST_SHIFT); // doesn't compile
// but the rest works fine...
}
So clearly the C syntax isn't simple. It's so not simple, that compiler writers can't agree on what exactly it is.(I've been writing such programs for the purpose of using them as edgecases to test my C compiler.)
Then of course you have the type declaration fiasco: <https://blog.golang.org/declaration-syntax>
Then there if of course the awful ternary operator.
Oh, and I almost forgot: spaces are insignificant... unless you're in a "#define" directive. Talk about language consistency...
Those things not only make life hard for compiler writers, but also for language users, because, I don't know about you, but when I write a program the smallest unit that I use to think about its source code is the token. When I can't easily predict tokenization, then something went really wrong. Same with other syntax complications. Guy Steele once said that when he designs a language he verifies that its grammar is LL(1) with a parser generator, which helps make sure that it's not only easy to write the parser and that parsing can be very efficient, but also that it's easy to understand by humans. C fails here big time.
The specific problem is this has two parses:
foo * bar;
It's either an expression multiplying foo by bar; or, if foo is a type, it's a declaration of a variable foo pointing to values of type bar.The normal way this is handled is to inform the lexer of known type names (e.g. by letting it peek at the symbol table) whenever it parses an identifier, so it can produce a type name token instead.
But generally I disagree that preprocessor is separate from C for the following reasons:
1. It is specified by the C standard.
2. Different C compilers implement it differently, just like they implement other parts of C differently.
3. There is some truth in the claim that it is a separate text transformation, but here I also do not agree entirely, because the C standard doesn't say that the preprocessor outputs text. It says that it outputs tokens and those tokens are then converted to tokens of C proper[1]. It is also clear that this model of "text transformation" is also not entirely what happens in real world compilers, because if it did we wouldn't get good backtraces for tokens that went out of the preprocessor after substitution. Maybe GCC serializes it to text at some point, but clearly it gets more information out of it than what a naive interpretation of the phrase "text transformation" would suggest.
[1]: Translation phase 7 of the C11 standard:
> Each preprocessing token is converted into a token.
The preprocessor is part of the language. Both in literal sense and in real-world. All C programs use the preprocessor. You need to deal with it. Also, the grammar of the preprocessor is intertwined with C proper due to "#if" directives. The expression on the right hand side of an "#if" is parsed according to the grammar of C proper.
> I'm not sure that anything you say about it is true (the preprocessor is complicated).
You can run the examples under compilers and verify. Also, the part about it being context-sensitive (with regard to tokenization in #include directives vs everywhere else) is explicitly noted in the standard. Nothing controversial here.
> Certainly, your first example won't compile without warnings for any sensible GCC options.
Compiles just fine without any warnings with GCC with options -Wall -Wextra.
I've written several C-style preprocessors. They can be fiddly to write, but you shouldn't think of them as lexing to the same lexemes as a C parser needs. You can take a lot of shortcuts there.
Things like '<' and '>' being overloaded is quite common, even in lexers. You see it with nested templates / generics in C++, Java, C# etc. There are simple tricks for dealing with the ambiguity between '>>' vs '>' '>' in 'T<U<V>>'.
(FWIW, having written some of those preprocessors, I think they're a bad idea. It's notable that other languages haven't copied the idea, other than C++ which inherited it.)
> Unlike most modern languages, which have a context-free grammar, it has a context-sensitive grammar.
Implies it C does not have simple syntax.
Is there a law a language with "simple syntax" cannot have context-sensitive grammar?
In fact, I'm not confused with simple function types. It is just that sometimes, complex function types are convolutions for me.