Zig cc: A drop-in replacement for GCC/Clang (2020)
andrewkelley.me
andrewkelley.me
https://dlang.org/spec/importc.html
It works by having an extra module called cparse.d
https://github.com/dlang/dmd/blob/master/src/dmd/cparse.d
which parses Standard C. The lexer and semantic routines of the D compiler are then tweaked to support C semantics. The same optimizer and code generator is used.
As a side effect of using the D compiler's innards, ImportC can do things like handle forward references, and even execute C code at compile time!
One assumes it is because C is so hard to rid ourselves of because POSIX (etc) that you're even entertaining this idea?
I mean, "it's fun" could also work as motivation, I guess.
I'm just curious!
There are some less theoretical things too, i.e. no more keeping kernel headers up to date (apart from ones that use inline ASM that we have to be careful with).
With ImportC, D will be one of the better and non-hackier ways to interface C with other languages - i.e. introspection and metaprogramming but in a real language rather than some brain-dead DSL
1. It's no longer necessary to translate .h files to D to interface with C. While doing the translations isn't hard, it is tedious, and if there are a lot of .h files it becomes a barrier.
2. .h files get updated over time, and correspondingly updating the D files is error prone. It's much more effective to just compile the new .h files.
3. It is not necessary to even create a .h file - just compile the .c file with ImportC and D will access it just like any other module. D will even inline the C functions in it.
4. Use it as a standalone, small and fast, C compiler.
P.S. I wouldn't be in the business if it wasn't fun.
I want to use SDL2 and cairo and pango? No problem! GTK4, easy peasy! There’s a C lib that comes as a single .c file that makes it easier to interact with SDL? Drop it on the project folder and compile it with Zig along all the other .zig files!
As somebody that has been avoiding C as much as I can, I’m loving this approach since it has open the doors to the world of C for me without having to learn autotools, make, valgrind, et al
Valgrind is also more useful than just for C, although admittedly it's most useful for binaries that have parts written in unchecked languages like C.
If you can't afford the virtualisation overhead, and you need to find leaks you should try what Raymond Chen suggests in "The poor man's way of identifying memory leaks" (not bothering to link since Microsoft will only move it anyway, they have several times since I read it). If you are too lazy to do it by hand, or find the technique works but wish it less manual, this is what Leakdice does:
https://github.com/tialaramex/leakdice OR https://github.com/tialaramex/leakdice-rust
(a site that tries to keep a proper backup of the ever moving writings of Chen)
If nothing else, I appreciate the advisement that it can have security implications.
You can't unleak secrets.
Using zig I finally decided to try GDB and it is amazing!
Compared to Zig, cross-compilation in C/C++/Rust seems downright user-hostile: you have to obtain the correct cross-sdk, but it won't tell you which one, or how to get it (some combos don't even exist), or even where to put it. If you don't guess everything correctly, it will fail with inscrutable errors, or worse, only fail at runtime on the target machine.
E.g. try to cross compile Metal shading example for iOS from Linux with zig, or do a Win2D UWP rendering application from Linux with zig.
And to go back to my examples, headers and libraries alone won't make it, without the remaining toolchain to create the appropriate package.
If you want to cross compile C stuff e.g. a Win32 GUI app with Direct X or get OpenGL bindings for SDL though those all work just fine if you provide the SDK files same as you would compiling natively. The advantage being now you can do all builds from a single build instance just by feeding it the libraries and a target platform.
The system API problem can be solved by shipping automatically generated API bindings (similar to what Microsoft does with the Rust bindings to the Windows APIs, a similar project for Zig is underway). Once those bindings exist, no Windows SDK is needed anymore and cross-compiling should "just work".
PS: Zig actually ships with the mingw headers in the default installation (e.g. windows.h, d3d11.h, dxgi.h etc are all there), I haven't actually tried to cross-compile a Windows executable using the Win32 APIs though, but it looks like it should work already.
Reverse engineering is legal in the US (unless bound by contract).
I don't see shipping all targets all the time to be a particularly scalable solution in general. When you start including all libraries necessary to develop actual apps, the download size will eventually get so big that people will complain. Just make it easy for developers to get the targets they need, but don't ship everything by default.
It seems to be what Flutter is doing, and it must be a ton of work.
For Mach (the Zig game engine I'm working on), I have a Zig build script which fetches the required parts of the macOS and iOS SDKs from GitHub to enable cross compilation from Windows/Linux -> macOS. Zig's linker, zld, enables that to work easily.[0]
For SDL.zig, they have also achieved cross compilation for most targets out of the box[1].
Contrast this to say, Go or Rust, where most GLFW and SDL bindings require mucking around with the system dependencies. It doesn't just work out of the box, I think that's really unfortunate and hope other languages work more towards painless installation of C-binding dependencies.
This has been the status quo for so very long. Usually these projects are maintained by different teams. IMO musl is more accessible than glibc and that's part of what makes it easier to distribute. Arm and x86 coexisting for as long as they have is a factor, and so is Kelley's ambition to make this use case well-supported. nostdlib is also well-supported by Zig btw.
I'd guess that a lot of Rust developers currently do development on their target platform, but that's because cross compiling is such a pain. If it were as easy as zig or go then things might well be differen.
My understanding was that most people were using "cross" docker images for "cross-compilation", because setting up native actual cross-compilation was quite involved.
Yes, people, including myself, use 'cross' precisely because it isn't as easy as "just use rustup to install the desired target." 'cross' has a bunch of Docker images that set up the cross compilation toolchains, and those images represent a fair amount of domain knowledge that I don't have to tediously acquire to use it.
I also think you're really missing out on the "compile C code easily too" thing. That isn't very pleasant at all in Rust today.
(Finally, one reason it’s so big is that we ship binaries. This ships source. std-aware cargo could really help here.)
So much this. Cross compiling pure Rust code is really easy today but as soon as you have a C dependency in your dep tree, you're gonna have a bad time.
That's not to say that it's impossible to do something like that in a C++ compiler but it's going to be more work than there is benefit. Tangentially related: https://twitter.com/__phantomderp/status/1416108650809241601
C++ also tries to bring in modern programming research when it makes sense, but that also explains why it's like an octopus with extra legs nailed on.
For example, move semantics are a peg leg for the octopus. Should an octopus have a peg leg? I'm not a biologist of any sort, but my guess is "No". I can see why C++ wanted to add move (without this performance in some scenarios is just awful) and yet since C++ didn't start out having move semantics and can't give up backward compatibility it has to shoehorn them in with this awkward "hollowed out object" idea. The result is performance still isn't where it could be and the feature is unergonomic.
It doesn't exist in Rust because nobody has proposed it or implemented it, simple as that.
I think developers can spare that much space. Besides, storage space is scaling much faster than the number of available targets, so I don't see how this could be "not particularly scalable" unless you either have a weird definition of "target" or predict there's going to be a sudden explosion in the number of targets.
Go also makes cross compilation to any supported target a breeze in the default toolchain download, and it is an incredible convenience for the developer.
I think a lot of the issue here comes down to targeting server workloads and command-line tools vs. apps using more of the OS libraries. In practice, Go focuses on the former kinds of apps, so shipping all targets is a more viable approach for them. But when you start getting into apps that want to use a larger swath of the platform facilities, then the size of the platform SDKs starts to become an issue.
I provided both Zig and Go as examples of what people are looking for, and their downloads are neither obscenely large nor do they include those massive SDKs.
So yes, you have a weird definition of "target" that no one else here is using, and it is disingenuous for the conversation. rustup can already add additional targets after installation, are you suggesting this downloads 10GB SDKs? Definitely not, and that sounds irrelevant. (Even once you do this, the toolchain experience is not as good as what Zig or Go offer.)
Compare what Zig and Go do to what Rust does, not some strawman argument that would require users to download 10GB of SDK with the Rust toolchain. Alternatively, find me someone who is saying they think these multi-gigabyte SDKs should be included, because I don't see that anywhere in this conversation.
People here are complimenting what Zig currently does, which takes less than 40MB compressed. Zig can cross compile to Windows just fine. There are limitations to everything, but what Zig and Go offer is strictly better than what Rust offers in terms of the out-of-box cross compilation experience, and it isn't unduly burdensome on the developers like your proposed 10GB download.
Until then, the lack of a great cross compilation experience out of the box is just a limitation of the Rust toolchain. It's an acceptable limitation in many situations, but I don't buy your repeated arguments on this HN discussion that it is unclear that this is desirable to fix. Clearly many people in this discussion alone disagree with your position, and my own anecdotal discussions with other developers in real life aligns with this discussion. YMMV, obviously.
I strongly disagree. These are different languages. Rust leans heavily on generics and monomorphization. It implements most language operations, like ptr::offset, in the language itself, increasing the size of library metadata. It supports tools like the sanitizers. It has a rich serializable MIR format so that generics can be embedded at a higher level than just machine code and a lower level than source.
It is absolutely not true that nobody "really wants" smaller binaries. Rust already did a fair bit of experimentation with running crate metadata through gzip, etc. years ago (turns out there are some thorny tradeoffs around compilation time vs. on-disk storage when you go that route). I can't speak to Zig, knowing less about it, but with Go there were conscious language design decisions that favor binary size over runtime performance (e.g. hash table lookups all going through a single function instead of being specialized). This is fine! But it's contrary to the idea that Rust could achieve smaller binary sizes if we "really" wanted to.
> Until then, the lack of a great cross compilation experience out of the box is just a limitation of the Rust toolchain.
Rust has a great cross-compilation experience. It's as simple as:
$ rustup target add arm-linux-androideabi
$ cargo build --target=arm-linux-androideabi
Except it's not the same. Zig also cross-compiles C code beautifully, and lots of Rust code depends on C libraries. That "one additional command" does not solve cross compilation to an equivalent degree as Zig.
Go code tends to avoid C because of the heavy penalty that CGo imposes, so the cross compiliation experience is usually good there, but for different reasons than Zig.
Cross compilation in Rust is better than C or C++. But it's still a big pain.
I really think you are underestimating what the Zig folks are doing. Please investigate more deeply.
In particular: "Zig does not ship with any pre-compiled libraries; instead it ships with source code, and builds what it needs on-the-fly."
But Zig’s approach isn’t actually about making everyone compile more things from scratch. It doesn’t compile libc from scratch any time you select a new target. Reading the article you can see the extreme lengths it goes to to avoid this.
The main thing that’s missing from Rust is target-specific libc.so/etc, and an appropriate linker for every target. If you try cross compiling anything, you will soon run into this problem; your system will not have the correct target libc or an appropriate linker. You can generally only change one part of the triple before the experience starts falling apart. I can cross compile for iOS/tvOS/etc on my Mac; that’s about it.
Zig manages to solve this for all its targets by pre-processing simplifications of the various libcs, which are then bundled into the Zig binary (as 3 very small files). It generates a useless but linkable .so file from the preprocessed files for any target on the fly. It gets you the correct libc headers. Then it uses LLD instead of making you hunt around on Ubuntu forum posts finding and downloading the correct GNU linker for your specific architecture. So you don’t link to a real libc, but you don’t have to compile one either. (Until you want to execute a binary with QEMU, but usually non-simulated target machines have a libc.so already.)
(Aside: Golang solves this by not depending on libc at all, by reimplementing most of it including the syscalls in Go, and I believe by using its own multi-target linker. Many pros and many cons, but an approach Google is happy to sponsor.)
If you’re saying this approach would be infeasible for Rust std/core, then yes obviously. It’s statically linked, you can’t get away with a fake .so file.
But that was never really the issue. Rustc can build std’s rlib files for any target effortlessly. And yet cross compiling is still a pretty poor experience. The state of the art for Rust developers at the moment is rust-embedded/cross, which solves the same libc+linker problem that Zig does, naively, by literally using Docker and per-target Ubuntu images to download prebuilt GCC and libc packages from Apt. Basically we can do way better than that, because Zig showed us how.
See this Dockerfile for what I mean: https://github.com/rust-embedded/cross/blob/master/docker/Do...
I'm honestly pretty disappointed at your participation in this thread. It seems like you're going out of your way to assume the worst possible interpretation of what folks are saying.
We could still do a few different things to make this story better for users, even if it's not literally identical.
Since advent of multiarch, crosscompilation does not seem harder than regular compilation. Just install package with gcc for target arch and required libraries for target arch (using the same way as gcc and libraries for host arch).
Lately I've cross-compiled a few times Linux/Windows programs, and the only thing I had to do (assuming that the underlying crates are compatible, and that the development libraries are in place, which is required for any language) was:
rustup target add x86_64-pc-windows-gnu
rustup toolchain install stable-x86_64-pc-windows-gnu
cargo build --target x86_64-pc-windows-gnu
There are surely issues with individual crates, and possibly with lower-tier targets, but to characterize this procedure as "user-hostile" doesn't make sense.
Rust totally craps out if you try to make a binary or dylib and you don't already have cross-compilation toolchain on your OS. It's doable from a Linux distro that has cross packages and a GNU target. It's a PITA for anything else, including anything from Windows and macOS hosts.
`zig cc` is actually useful even for cross-compiling Rust projects, because some Rust crates depend on C libraries.
pip install ziglang
python3 -m ziglang cc my_c_prog.c -o my_c_prog
./my_c_prog
But what surprised me immediately and the thing I'm absolutely loving is the error handling (https://twitter.com/mgill25/status/1416720958988210177) and the comptime features.
I've done Rust and while I love what it's doing, getting into low-level C-world without the overhead of a language like Rust feels liberating.
Zig is 90% of the way there with error handling, but it is extremely obvious to me that you need to be able to have error context carried with the error... it is apparently not obvious to the core Zig team[0], and this is a major impediment to my interest.
Sure, if you're compiling to a microcontroller, maybe you're willing to sacrifice usability to avoid error values ever allocating... but most developers can spare an allocation for errors, especially given that error handling is generally not the hot path.
[0]: https://github.com/ziglang/zig/issues/2647 (among other issues)
While you can argue (as people do on that issue) that there are workarounds to carry error payload, the truth is if you don't make it ergonomic, many libraries will not do it, and that will filter down into software using those libraries, thus hurting the final resulting software.
I'm hopeful this feature will eventually get added.
I think it really is not obvious once you start thinking about all the implications. Right now the recommended way of reporting extra information alongside errors is by using a "diagnostics" struct.
You can see an extremely effective example of that in zig-clap, for example.
https://github.com/Hejsil/zig-clap
Also this pattern was also mentioned in the issue you mentioned: https://github.com/ziglang/zig/issues/2647#issuecomment-5898...
The advantage of this system is that the caller can decide whether error diagnostics are desirable or not. If they're not, then zig-clap doesn't waste cycles nor memory keeping track of the context in which an error happened. It's the best of both worlds in my opinion.
No... it really is obvious. I've written quite a bit of C, C++, and Rust over the years, in addition to all sorts of garbage collected languages. I'm not writing this as an off-the-cuff thought. I've even written a similar comment before on HN about Zig's error handling. I've thought about this for a long time. It is just as obvious now as it was on day one.
The fact that the Zig team keeps resisting this is just mind blowing to me. "Penny wise, pound foolish" is my opinion. The actual runtime cost here would be so entirely negligible, but its absence is a major limitation.
> The advantage of this system is that the caller can decide whether error diagnostics are desirable or not. If they're not, then zig-clap doesn't waste cycles nor memory keeping track of the context in which an error happened. It's the best of both worlds in my opinion.
Instead, the developer has to waste mental cycles and memory keeping track of the context on every single function call. Strongly disagree on it being the best of both worlds.
For starters, every library has to decide whether they will support it or not. If the language doesn't make this a built-in part of the core error handling, then spoiler: they won't. They'll throw away all useful error context, leaving no option for the application developer to add it back. Once the context is gone, it's gone for good. You can't know what a library did to cause an AccessDenied error... you can't reach into the library and pull that context out of thin air. If every library is implementing this diagnostics pattern, then that's an equally strong argument for it to be supported at the language level. Either context is being irrevocably lost or developers are wasting time writing convoluted code. Either way, this is not the optimal solution.
I have experience writing code in a lot of languages... error context is extremely important, and the "waste" of CPU cycles and memory is extremely minimal for the benefit gained.
As I have hinted before, a compile-time switch to throw this information away could be a valid solution for the extremely-difficult-to-imagine situations where great error handling is unacceptably costly on performance... then it would be like Zig's "async" support, where the library developer doesn't have to color their functions based on how the end user will use it. They should return all the error information all the time, and the compiler decides whether to keep the context or throw it away and provide only the error "name". Yes, this would bring its own challenges, but seriously, this option to discard the context is completely unnecessary outside of like 8-bit microcontrollers, and probably isn't worth implementing. Just provide context all the time. Please... it's such a small ask.
Looking at the links you've provided, which you have also provided in past discussions here, that pattern is extremely painful. I cannot overstate how much worse that usability is compared to the regular error handling Zig provides, and so obviously no one will ever use it. It serves as a pattern for the most desperate situations, and as a way to handwave the lack of language-level support for error context, in my opinion.
84 upvotes for the initial issue is substantial in a community the size of Zig.
For especially difficult language design decisions, I like to work on other language changes that I am more confident about as a way to anchor the design, and then revisit the difficult questions, hopefully armed with more guidance from the rest of the language design being in place.
No need to install yet another toolchain, this is just another target Zig can compile to, out of the box.
Emscripten just installs a bunch of stuff to make it easier to use existing libraries.
If you just want to use standard library, no need for extra stuff.
Zig can target both without having to install anything else.
Zig packages them, naturally you don't need to install them a second time.
>The major theme of the 0.9.0 release cycle will be stabilizing the language, creating a first draft of the language specification, and self-hosting the compiler.
So I am guessing we could see 1.0 release within 12 months time? Or will there be 0.10 or 0.11 after that?
I am just eager to hear people's comments and opinions for using it in production within their context and edge cases.
And as a sort of side note, I quite like the momentum and the way Andrew has been handling Zig's PR. I just hope and pray HN keep that tradition, dont over hype it.
[1] https://ziglang.org/download/0.8.0/release-notes.html#Roadma...
1.0 is going to take a lot longer than that.
> You can download a 45 MiB tarball, unpack it, and you're done. You can even have multiple versions at the same time, no problem......Compare this to downloading Clang, which has 380 MiB Linux-distribution-specific tarballs. Zig's Linux tarballs are fully statically linked, and therefore work correctly on all Linux distributions. The size difference here comes because the Clang tarball ships with more utilities than a C compiler, as well as pre-compiled static libraries for both LLVM and Clang. Zig does not ship with any pre-compiled libraries; instead it ships with source code, and builds what it needs on-the-fly.
We hope to be using TigerBeetle which is written in zig in production in 2022: https://github.com/coilhq/tigerbeetle
$ alias zcc='zig cc'
shopt -s expand_aliases
The latest proposal from andrewrk is a new zig subcommand which would automatically generate wrappers ("zig-cc", "zig-c++", etc) for for the target, then set them in the current env ("CC=zig-cc", "CXX=zig-c++") [1].
zig cc is really, really cool, and it's incredibly generous that it provides zig's benefits to users of C and C++ as well.
With GCC and Clang, I only pay for cross compilation if I need it, so for a fair comparison, the cross-compiler libraries should be installed for GCC and Clang. Also with GCC and Clang, I don't have to worry about a third party (the Zig team) incorporating bug and security fixes from upstream. And with GCC I get support for more platforms.
If I have chosen to use cross-compilation, building for alternative architectures is just as easy as for Zig.
Yup! https://github.com/ziglang/zig/commit/db17c0d88c44183b306099...
> Also we build LLVM tip of tree from scratch...
I would have also thought it would be non-trivial (Julia uses LLVM and has some non-backwards compatible patches) but this paragraph intrigued me:
>> At the time of this writing, LLVM 10 was just released two hours ago. It will take days or weeks for it to become available in various system package managers. But you can already download a master branch build of Zig and play with the new features of Clang/LLVM 10. For example, improved RISC-V support!
I found this for LLVM 12 (released April 14) https://github.com/ziglang/zig/commit/5a3ea9beced660c9cc463b... https://github.com/ziglang/zig/commit/e248de93a07fd6a7f48581... https://github.com/ziglang/zig/commit/6a7f21d1b8ca5c082c804e... https://github.com/ziglang/zig/commit/1438c324a460789f1004a3...
Andrew seem to be following along their release pretty closely.
Disclaimer: I don't know squat what I'm talking about
Who knows, though. Someone might end up implementing a C compiler on top of the self-hosted Zig compiler's backend!