Distcc: A fast, free distributed C/C++ compiler | Hacker News Reader

Distcc: A fast, free distributed C/C++ compiler

distcc.org

220 pointseating55599 comments

Hi, distcc's original author here. It's really nice that people are still enjoying and using it 20 years later.

I have a new project that is in a somewhat similar space of wrapping compilers: https://github.com/sourcefrog/cargo-mutants, a mutation testing tool for Rust.

Pretty sure I used cargo-mutants on a lark during Advent of Code a couple years ago. Caught a couple bugs with it. Good stuff. Thanks! :D

Wow, it's already 20 years. Was setting up distcc in my first job, with ccache for armcc. Good times, great experience. This was a life-safer in terms of compilation speed (just needed wait 20 mins for consolidation and like the same or more for loading symbols to the debugger, which was bearable comparing to spending another 2 or 3 hours taken by compilation). Having the opportunity, would like to thank you!

Hey, I just would like to let you know that DistCC always has a special place in the hearts of loyal Gentoo users like me :) Kudos!

https://github.com/icecc/icecream - another option that does what distcc does, but aimed at a somewhat different use case.

https://ccache.dev/ - a similar idea but provides caching of build outputs instead of distributing builds. You can use it together with distcc to achieve even better performance.

ccache is used together with distcc at the current place I am working at. Started digging at how these two work as I thought there is still room for improvement in our build times that can vary between 10 minutes to 1 hour. It is a huge code base, easily more than a million lines and around 18k files. But had to stop as there were way too many features to develop and bugs to fix. Also, management does not see that kind of work as useful so no point fighting those battles.

You are probably aware, but for. others with ccache this is called "cache sloppiness", which is my favourite term.

You can set this via config, as by default ccache is paranoid about being correct. But you can tweak it with things like setting a build directory home (this is great for me, as I'm the only user but compile things in say `/home/josh/dev/foo` and `/home/josh/dev/bar` and have my build directory as my dev directory and it's shared. (see https://ccache.dev/manual/latest.html for all the wonderful knobs you can turn and tweak).

Fantastic tool, the compression with zstd is fantastic as well.

I played with distcc (as I have a "homelab" with a couple of higher end consumer desktops), but found it not worth my time as compiling locally was faster. I'm sure with much bigger code bases (like yours) this would be great. Reason I used it it that archlinux makes it super easy to use with makepkg (their build tool script that helps to build packages).

The best use of distcc "at home" is when you have one or more "big iron" (desktop, server, whatever) and a few tiny machines that work just fine but don't have much processing power.

For example, with some work, you can setup distcc to cross-compile on your amd64 massive box for your raspberry pi.

For builds that large, I (personally) start evaluating Bazel. Bazel has distributed build + shared cache features built-in. But I’ve always just dug into reducing build times in any large C or C++ code base I’ve worked on—damn what management says is important. And the switch to Bazel can be costly (effort) and it may be difficult to get team buy-in.

My codebase is significantly larger than yours (mine's a mix of mostly-C++ & some C) — perhaps 10–12 million lines. Clean builds are ~10m; clean-with-ccache are ~2m; incremental are millisecond.

I know this probably won't help with your current project, but you should think of your compiler as an exotic virtual machine: your code is the input program, and output executable is the output. Just like with a "real" CPU, there are ways to write a program that are fast, and ways to write a program that are slow.

To continue the analogy: if you have to sort a list, use `qsort()`, not `bubble sort()`.

So, for C/++ we can order the "cost" of various language features, from most-expensive-to-least-expensive:

    1. Deeply nested header-only (templated/inline) "libraries";
    2. Function overloading (especially with templates);
    3. Classes;
    4. Functions & type definitions; and,
    5. Macros & data.

That means, if you were to look at my code-base, you'll see lots-and-lots of "table driven" code, where I've encoded huge swathes of business logic as structured arrays of integers, and even more as macros-that-make-such-tables. This code compiles at ~100kloc/s.

We don't use function-overloading: one place we removed this reduced compile times from 70 hours to 20 seconds. Function-overloading requires the compiler to walk a list of functions, perform ADL, and then decide which is best. Functions that are "just C like" require a hash-lookup. The difference is about a factor of 10000 in speed. You can do "pretend" function-overloading by using a template + a switch statement, and letting template instantiation sort things out for you.

The last thing is we pretty much never allow "project" header files to include each other. More importantly, templated types must be instantiated once in one C++, and then `extern`ed. This is all the benefit of a template (write-once, reuse), with none of the holy-crap-we're-parsing-this-again issues.

I love your comment and it is 100% spot-on. extern C is the magic sauce for making anything fast.

The only downside is that it adds a ton of boilerplate and a lot of maintenance overhead. You need separate compilation units for everything and then you need a sub-struct to use the pimpl approach. Fast pimpl (in-place new in reserved space in the parent struct itself) gets rid of the heap allocations but you still have a pointer indirection and prevent the compiler from properly stripping out unused code across translation units normally (that’s where LTO comes in these days).

Really, the problem is just that it’s a PITA to write compared to sticking everything in the header file.

(It’s ironic that rust meets the first two rules by design but is still much slower than C++ to compile, though it does imply what’s already known, specifically that there’s a lot of room for improvement.)

https://github.com/mozilla/sccache is another option which addresses the use cases of both icecream and ccache (and also supports Rust, and cloud storage of artifacts, if those are useful for you)

The maintainers of icecream and distcc have an understanding that we want to bring the two back together. It will probably never happen as nobody wants to do the work (and there may be some differences in how they work that are not resolvable)

Back at Uni (two-and-a-half decades ago now), I setup a hacky version of this sort of thing to distribute compilations over several workstations:

* use make -j to run multiple task at once

* replace¹ gcc with a script that picked a host and ran the task on that²

This had many limitations that distcc lists it doesn't have (the machines had to have the same everything installed, the input and output locations for gcc had to be on a shared filesystem mounted in the same place on each host, it only parallelised the gcc part not linking or anything else, and it broke if any make steps had side-effects like making environment changes, etc.) and was a little fragile (it was just a quick hack…), but it worked surprisingly well overall. For small processes like our Uni work it didn't make much difference over make -j on its own³ but for building some larger projects like libraries we were calling that were not part of the department's standard Linux build, it could significantly reduce build times. On the machines we had back then a 25% improvement⁴ could mean quite a saving in wall-clock time.

One issue we ran into using it was that some makefiles were not well constructed in terms of dependency ordering, so would break⁵ with make -j even without my hack layered on top. They worked reliably for sequential processing which is presumably the only way the original author used them.

----

[1] well, not actually replace, but arrange for my script to be ahead of the real thing in the search path

[2] via rlogin IIRC, this predates SSH (or at least my knowledge of it)

[3] despite the single-core CPUs we had at the time, task concurrency still helped on a single host as IO bottlenecks meant that single core was less aggressively used without -j2 or -j3

[4] we sometimes measured noticeably more than that, but it depended on how patballisable the build actually was, and the shared IO resource was also shared by many other tasks over the network so that was a key bottleneck

[5] sometimes intermittently, so it could be difficult to diagnose

> some makefiles were not well constructed in terms of dependency ordering, so would break⁵ with make -j even without my hack layered on top

That remains true to this day of even makefiles from some really big name projects. Huge pet peeve of mine and I’ve filed PRs with fixes many times.

Nearly twenty years ago I had a little server farm of old PCs. Two or three Pentium-133s, one dual Pentium Pro 200 machine, and my pride and joy, a Pentium 3 running at 600 MHz. I was trying to get familiar with Gentoo and to make recompiling everything all the time more bearable I set up distcc so my P3 could do most of the work. It worked very well!

But after a few weeks every Gentoo box in the house started crashing regularly. It took me a while to figure out what was going on: one of the slower machines had developed a single-bit memory error and was sharing corrupted .so files with all other machines.

Yeah. Everyone I’ve talked to who has run a build cluster has recommended ECC for the build cluster, even if they’ve decided not to use ECC for other systems. Some people would run Hackintosh-like setups for macOS build clusters, just for the ECC.

Reproducible builds are also a big win here.

Absolutely! This was a very educational experience.

I still need to get around to setting up distcc; I only have two Gentoo servers, but one is so much more powerful than the other, and their CPUs are close enough I might be able to use ccache, too ...

> their CPUs are close enough I might be able to use ccache, too

It’s the compiler that needs to line up for that. But my recommendation is to install sccache which will figure it out for you.

I had to break into a laptop for my dad last week. Last used 7 years ago. Running win2k on a pentium 3.

Had to jump through various hoops and find creative ways around problems (laptop too old to boot from USB, all my modern machines lack optical drives. And so on) but I did eventually break in.

25+ years ago, our company used Clearcase for version control and it's clearmake had distributed build capability. Clearcase used a multi version file system (MVFS) and had build auditing so clearmake knew exactly what versions of source files were used in each build step. It could distribute build requests to any machine that could render the same "view" of the FS.

Even without distributed builds, clearmake could re-use .o files built by other people if the input dependencies were identical. On a large multi-person project this meant that you would typically only need to build a very small percentage of the code base and the rest would be "winked in".

If you wanted to force a full build, you could farm it out across a dozen machines and get circa 10x speedup.

Clearcase lost the edge with the arrival of cheaper disk and multi-core CPUs. I'd say set the gold standard for version control and dependency tracking and nothing today comes close to it.

Brings back memories. Fresh out of college, I was given the additional job of being the Clearcase and Unix admin for my team. Not that I had any special skills but others didn't know a few Unix commands (System V) that I did. But Clearcase was such a good product and was used in the Telecom companies that I worked for (Motorola, Lucent etc.) It was owned by Rational at that time and if memory serves me right, were acquired by IBM.

To this day, I find Clearcase's way of doing things is the better way to do version control. Git, in comparison, kind of feels alien and I could never really get the same type of comfort on it.

That was also the standard development workflow at Nokia Networks, back when NetAct was being developed for HP-UX.

Nowadays most of it has been ported into Java, if the ongoing efforts when I left, finally managed to migrate everything away from C++, Perl and CORBA.

Clearcase was utter crap. 6 hour code checkouts and 2 weeks to setup a new developer is a freaking joke. I literally did a conversion from Clearcase to git and reduced the setup time to 15 minutes and this is for a code base older than Clearcase is.

Not to mention the absolutely bad design for handling merge conflicts (punt to human if more than 1 person touched a file seriously???)

If you're talking about Clearcase snapshot views, I agree they were garbage. And IIRC merging in a Clearcase snapshot view was also a hot mess. Snapshot views was a bold-on that we were forced to use in later years. TBH the migration to other VCSs was already underway by then in our company but snapshot views was the last straw for us.

On the other hand Clearcase dynamic views were pretty awesome. You just needed to edit your view config spec and the view FS would render the right file versions immediately. No checkout required. There was even view extended naming to let you open multiple versions of the same file directly from any editor.

As for merging Clearcase set the gold standard for three-way automatic merges and conflict resolution that wasn't matched until git came along. It's still superior in one important way - Clearcase had versioned directories as well as files so you could commit a file move and someone else's changes to the same "element" in the original file location would be merged correctly after the move. No heuristics, just versioned elements with versioned directory tree entries. Backporting fixes was a breeze, even with a significant refactor in between.

Git more or less wins hands down today because it is fast, distributed and works on change sets. But something with git's storage model, a multi version file system and Clearcase's directory versioning would be an awesome VCS.

we got a bit closer with gitfs but nobody has really merged all the parts into a "it just works" setup.

https://wiki.archlinux.org/title/Gitfs

Around that time I went down the rabbit hole of "what if I wanted to write a filesystem for Windows NT?" It turns out that, aside from network redirectors, Microsoft didn't think this was something people ought to do very often. The people at Atria (which got bought by Rational) who were working on ClearCase were pioneers, and they bought a source license from Microsoft to be able to do it. The result was that there was a lot of information from them on a mailing list about how do implement a filesystem. However, it was mostly incomprehensible to me, because kernel programming on NT is quite different from what one would expect coming from other backgrounds, including Windows 95, OS/2, and Linux.

Clearcase's (clearmake's) cloud building capability sounds nice, but I have to pile on to this:

> I'd say set the gold standard for version control and dependency tracking and nothing today comes close to it.

In my 2005-2011 experience with Clearcase, it was slow and required dedicated developers just to manage the versions, and I'm so happy its version control model has been an evolutionary dead-end in the greater developer community. The MVFS is an attractive trap. Giving people the ability to trivially access older versions means you've just outsourced the job of keeping everything working together to some poor SOB. It was very much a "enough rope to hang yourself" kind of design.

As I said, it was slow, because MVFS. The recommended solution from Clearcase/IBM was to break up the source tree into different volumes (or whatever Clearcase's "repo"-analogue was named), which just increased the pain of keeping things in sync.

Additionally, it was an "ask-permission" design for modifying files, where you had to checkout a file before being able to modify it, and you couldn't checkout if someone else had, which added a ton of painful overhead.

I'll grant that my company/group didn't know what they were doing, but following IBM/Clearcase's guidance was not a good idea.

These days, I use Clearcase as a warning to the younger generation.

Fastbuild https://www.fastbuild.org/docs/home.html is the free distributed compilation system many game companies use. The combination of automatic unity builds (simply appending many .cpp source files together into very large combined files), caching and distributed compilation together gives you extremely fast C++ builds.

Also supports creating project files that are compatible with XCode and Visual Studio so you can just build from those IDE's and a pretty flexible dependency based build file format that can accomodate any kind of dependency.

Someone just linked Fastbuild few days ago in one community i follow and i did take a look as i havent heard about that project before. I didnt really like the idea that it would need to have its own build rules but its understandable - where as distcc would just integrate directly to what ever build tool you would be using. Also the syntax didnt look so "pleasing" but i guess the behefits can out weight the learning curve of yet an another build language (compared to meson/bazel/cmake et al) ..

Having distributed builds on msvc thought with free software sounded very promising thought.

Game companies didn't hear of ccache then.

Isn't it a misnomer to call it a "compiler"? Even it's github README says otherwise,

> distcc is not itself a compiler, but rather a front-end to the GNU C/C++ compiler (gcc), or another compiler of your choice. All the regular gcc options and features work as normal.

I guess it depends. You use this software to convert your source code into the binary blobs efficiently. From high level, it behaves like a compiler, but internals are just using other compilers.

A lot of other compilers are the same way. gcc and g++ are higher level wrappers around their low level compiling and linking executables. It depends how you define the terms, including what you mean by "program".

> You use this software to convert your source code into the binary blobs efficiently.

So `make` is also a compiler?

20+ years ago I used distcc + jpegtran to rotate a few thousand jpegs. From what I remember I just had to write the Makefile as you would normally.

I think the end result was that the transfer time at 10mbps speeds made it take longer than it would have just transforming locally, but it was neat to see it work!

We used it at a previous job. Every dev computer in the building ran distcc, compiles were distributed and finished super fast.

Also had much success with it in the past. In my previous job we wanted to use it when building Xcode projects with a large C++ library. However, it's not compatible with Apple's modified Clang. I spent some time maintaining a fork where I hacked in support for passing along the Apple specific switches but eventually gave up. Seems like a cool feature would be the ability to define non-standard switches in a config rather than have them hard coded.

edit: just remembered I wrote up a blog post about it when I worked there

https://pspdfkit.com/blog/2017/crazy-fast-builds-using-distc...

Good gracious. Just looked at CHANGELOG. It was 20 years ago this month that I made modest code contribution to this project.

That's the beauty of low level code... It can last generations!

Also see, "code that works"

We used to use this in a previous company. It reduced the build time from ~20 minutes to ~2 minutes; until a bug in the ld linker at the time (linking is not distributed - only compilation is) pushed it back up to 20 minutes. We had a hell of a time finding the issue, but luckily there was a newer versions of binutils available where it was fixed; and upgrading to it got the build time back under 2 minutes.

Fun times!

Some other multi machine options that have worked well for me, well beyond just compilation of C/C++ on multiple machines, with multiple gpu(s) and cores.

1) set up passwordless, ssh.

and

2) use the gnu parallel. https://www.gnu.org/software/parallel/

gnu parallel is super flexible, very useful.

So many memories. Back in grad school I would run this on a handful of workstations to speed up compilations. Really neat tool, and a clever setup.

Back when I ran Gentoo, distcc and ccache were pretty a necessity.

Oh yeah, I think I may have found distcc by way of trying out Gentoo. It makes sense. It's been a while since I heard about them!

I really prefer icecream to distcc: it handles toolchain distribution, scheduling, and discovery.

This. If you're thinking about distcc, try icecream first, it is much nicer.

> But unlike distcc, Icecream uses a central server that dynamically schedules the compile jobs to the fastest free server.

https://github.com/icecc/icecream

I'm curious about the security implications with using distcc. Doesn't this mean that if one computer gets compromised, the attacker can run code on all other computers using distcc, or secretly inject malicious code in the build result.

So using distcc means that all computers using it must be trusted. And that means that using it on "all developers computers to share the load" is good for performance but bad for security.

Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach.

I'm not sure whether distcc affects reproducible builds?

You could, in any case, have tighter controls on the release builds, which would be done on a CI machine before signing.

(Back when I used distcc we didn't distribute across the dev machines, we had an entire build farm of two racks of 2U servers!)

> Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach.

I would not generalise so quickly. Is every computer compromised in the internet if one is compromised?

No. It highly depends on the trust between those machines and whether they share similar services with critical vulnerabilities. Only then, they might be compromised together.

But the world has evolved and not everyone anymore bases their total trust and security thinking for "no outside internet connection, we are fine".

> Is every computer compromised in the internet if one is compromised?

Internet is no lan (local area, L2) where computers do indeed typically have more generous policy regarding access between each other.

Think about the windows firewall asking whether you just connected to a work/public/home network

> Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach.

That might've been true 10 years ago and still is in some case but I wouldn't assume that now, far more things run over encryption for example, or have firewall

I don't know about other platforms... but on Gentoo, distcc is restricted to only running a small list of programs (ie gcc). That means you can't just send random commands to the distcc host from your "slow machines".

Granted, if your distcc host gets compromised, the compiled output shouldn't be trusted until the server can be reprovisioned.

Yes, that's correct: reusing developer's computers means you're trusting all of them. At small scale, that may be acceptable because you probably trust them all a lot anyhow, and perhaps are unlikely to have a strictly hermetic and isolated build system.

When the team is bigger or when security is more important, it's important to have a build system where you're confident that no one can subvert the output, and that includes ensuring that very few people can control machines running distccd.

Another way in which it shows its age is that, by default, there are only netblock based restrictions on clients, and connections are over unencrypted TCP by default (last time I looked), although there is an option to use SSH (or I guess Tailscale or similar.)

IIRC, when using distcc you must make sure that the Toolchain is ABI-perfect identified by its path.

So:

/opt/ourcompany/dev/bin/cc

absolutely *needs* to be the same on all machines involved or you risk very hard to spot issues.

For that reason, either use the absolute same distribution for everyone (and then run /use/bin/cc but watch out for alternatives!) or roll-out your own toolstack but make sure to put its version in the path.

Sounds like a great use case for containers, no?

Maybe, but you need to run the exact came container on everyone's machines.

While it's purpose is different it can be used to do distributed compiling, so I'll leave it here.

https://github.com/Overv/outrun

Since I was just going down this rabbit hole recently, I kind of wonder if it's possible to set the filesystem on something more like the BitTorrent protocol so things like the libraries/compilers/headers that are used during compilation dont all need to come from the main pc. It probably wouldn't be useful until you reached a stupid number of computers and you started reaching the limits of the Ethernet wire, but for something stupid that can run on a pi cluster it would be a fun project.

How do these distributed build tools work across the Internet, i.e. outside of a local LAN office setting? Does the increased latency and slower bandwidth become a bottleneck?

Assuming you don't use NFS to continuously read and write, I think it would be fine. Compiling is generally speaking CPU bound. Even the website mentions that it just copies all preprocessed code to the workers, so I expect this to be a one-time overhead per job.

If you distribute preprocessed code, you need LAN speeds/latency for it to be faster than a local compilation. Otherwise, you need to have the exact same headers in addition to the exact same compiler installed everywhere. It MAY work otherwise, until it breaks in a subtle / obscure way one day and after tracking it down you will think maybe compilation doesn't take THAT long on a single machine :)

So all in all, I wouldn't say works outside of a well-controlled cluster on a LAN.

They will work over the internet. You should use SSH or a VPN like Tailscale because the native protocol is not encrypted.

Bandwidth and latency effects will have some effect on how well it will scale, but on a reasonably high latency and low RTT connection it's probably still faster than purely local compilation. Sending work over the network takes time but not much CPU, so the local CPU is freed up to do other work.

Good memories! I had fun with distcc by compiling kernels across a few local machines back when desktops for mere mortals were dog slow, and it helped a lot. I never used it for cross compiling though, which is something that could help today when starting compilations from small embedded boards. Did anyone have success in a mixed environment, such as a small ARM board with native GCC plus one or more faster x86 machines with cross compiling tools installed?

Just tried this. I have a slow ARM laptop and a fast x86 desktop. The ARM laptop seems to detect the desktop as specified in the environment variable, but it doesn't accelerate compiling at all. The x86 box sits idle while the ARM laptop compiles. I have the aarch64 cross compiler installed on the desktop as well. Maybe I'm just bad at RTFM, but I can't seem to get it to work.

I don't have enough details to debug, but something I discovered just the other day might be helpful if you're building the kernel. For kernel builds, you have to specify CC= after `make`, not before. E.g. `make -jX CC="distcc alpha-unknown-linux-gnu-gcc"`.

Is there a reason to uses this over something like bazel build?

It works transparently with any project using autotools/cmake/meson. (which is 95% of open source projects, and most likely most enterprise software) No need to convert your project to starlark BUILD files.

Is bazel build distributed? distcc lets you use a lot of machines to speed up the wallclock-time of your compilation.

> Is bazel build distributed?

Yes, that's the main feature of Bazel. And it caches the generated files. Although you could do theoretically cache using ccache with distcc.

I doubt that it is the main feature. RBE is difficult to deploy and consequently rarely used. On the other hand of you assume that 99% of bazel invocations are done by google, then weighted that way RBE is nearly universal.

It can be. By default it is local.

But it has protobufs interfaces (IIRC), so a distributed build farm would generate the grpc endpoints for their implementation and then you tell bazel on the command line (or via .bazelrc) the address of the build farm it can use.

There's a couple of projects that implement the distributed/grpc part, the main one is https://github.com/bazelbuild/bazel-buildfarm

Not only it's distributed like distcc, Bazel also provide sandboxing to ensure that environment factors does not affect the build/test results. This might not mean much for smaller use cases, but at scale with different compiler toolchains targeting different OS and CPU Architecture, the sandbox helps a ton in keeping your build result accurate so that it could be cached remotely (like using ccache).

On top of it, the APIs Bazel uses to communicate with the remote execution environment is standardized and adopted by other build tools with multiple server implementation to match it. Looking into https://github.com/bazelbuild/remote-apis/#clients, you could see big players are involved: Meta, Twitter, Chromium project, Bloomberg while there are commercial supports for some server implementations.

Finally, on top of C/C++, Bazel also supports these remote compilation / remote test execution for Go, Java, Rust, JS/TS etc... Which matters a lot for many enterprise users.

Disclaimer: I work for https://www.buildbuddy.io/ which provides one of the remote execution server implementation and I am a contributor to Bazel.

It's a shame that distcc and ccache aren't more well known. Ccache, in particular, saved me probably years worth of compile time working with Qt.

Ooh this brings back memories. This project made my embedded cross-compilation iteration speed go from like 2-30 minutes to 5. This was 20 years ago. I don't understand why this isn't more of a thing these days. I think Bazel can do this, but I've felt nothing but pain from using Bazel.

Can anyone explain the architecture/how it works at a high level? I get that it is distributed. Does it basically copy the complete source tree to every worker and have them compile some independent subset of the object files? Does performance scale linearly with the number of worker nodes?

Per the description:

> distcc sends the complete preprocessed source code across the network for each job, so all it requires of the volunteer machines is that they be running the distccd daemon, and that they have an appropriate compiler installed.

So all the "environment" is on the source machine and just a bare compiler is required on the remote machines for compilation.

Last time I looked, it basically ran per-file "cc -E" on the source machine to get a compilation unit (optionally checking for a ccache cache hit at this point), then piped the result to "cc" running on the target machine, and copied the resulting object file back.

> Does performance scale linearly with the number of worker nodes

Yes, for small N.

Overall scaling was limited by how much "make -j" the source machine could cope with.

This was the original approach, although later work added an optional "pump mode" in which headers are distributed: https://manpages.ubuntu.com/manpages/bionic/man1/distcc-pump...

It runs the preprocessor locally, then sends that out to a volunteer node. https://www.distcc.org/distcc-lca-2004.html:

“The client is invoked as a wrapper around the compiler by Make. Because distcc is invoked in place of gcc, it needs to understand every pattern of command line invocations. If the arguments are such that the compilation can be run remotely, distcc forms two new sets of arguments, one to run the preprocessor locally, and one to run the. compiler remotely. If the arguments are not understood by distcc, it takes the safe default of running the command locally. Options that read or write additional local files such assembly listings or profiler tables are run locally”

Scalability:

“Reports from users indicate, distcc is nearly linearly scalable for small numbers of CPUs. Compiling across three identical machines is typically 2.5 to 2.8 times faster than local compilation. Builds across sixteen machines have been reported at over ten times faster than a local builds. These numbers include the overhead of distcc and Make, and the time for non-parallel or non-distributed tasks”

from section 3 (Design) of https://www.distcc.org/distcc-lca-2004.html

'''

distcc distributes work from a client machine to any number of volunteer machines. (Since this is free software, we prefer the term volunteer to the slaves that are used by other systems.)

distcc consists of a client program and a server. The client analyzes the command run, and for jobs that can be distributed it chooses a host, runs the preprocessor, sends the request across the network and reports the results. The server accepts and handles requests containing command lines and source code and responds with object code or error messages.

'''

> distcc sends the complete preprocessed source code across the network for each job, s

I assume that for template-heavy c++, that could easily be hundreds of gigabytes for 1 gigabyte of c++ code to compile...?

If you're working from a laptop, surely the wifi connection will by far be the bottleneck?

It is. These systems (inctedibuild, fastbuild) tend to work extremely well but are network bottlenecked. You need a fast, low latency connection.

I used this in a uni lab (good lord 15 years ago) when we needed to compile for a robot. It took maybe an hour or two to compile run directly on the robot but I was able to setup distcc to run on all the lab machines in that room and get things done fast.

Used distcc back in mid-2000s to crosscompile NetBSD kernel/userland for a TS-7200 SBC ARM device. Back then those devices didn't have much processing power, would have taken days otherwise.

Wow, it's still active! I remember when MBP wrote it while we were working at OzLabs together.

One day I'll get back to that rewrite of ccontrol using modern distcc's features...

Hi Rusty!

For a later hack in the same vein, check out https://github.com/sourcefrog/cargo-mutants

Title may be a bit confusing since skimming the project's homepage reveals that it's not a compiler, just frontend for other C++ compilers.

The biggest problem with distcc is that it fails with architecture-specific instruction sets like avx.

A problem not shared by icecc.

Isn’t that just a matter of setting appropriate target flags, and not letting them by implied? That’s basic build hygiene anyway.

Had no idea about this tech, very cool!

This is such a nice tool, I use it in my arch linux machines, it even supports cross compiling!

Xcode used to have native support for this, long ago. And it had mdns support too!

I've used it for remote raspberry pi compilation. Works very nicely.

storage becomes much faster than network today, so is it still worth it ? (naive question sorry) or maybe the cpu demand is much bigger than IO demand ?

Used to use this with Gentoo when doing emerges. Reusing old P3s and stuff, those were the day..

Insert XKCD "Compiling..." meme :)

Same, whole office stringed together so I could compile gentoo on my 700MHz laptop in reasonable time.

Reply on news.ycombinator.com