Hacker News Reader

Comment by pjc50 | Hacker News Reader

The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".

View on HN

Google has objfs, a authenticed network mounted drive that is an object file cache. Usually most of the tree has already been compiles so just your change needs to get recompiled and relinked.

Further, there's then GOMA which is the distcc equivalent. I frequently build Android with 500 cores in the cloud. For that reason, I don't think Google engineers will ever focus on the compiler speed of llvm. That said, LLD (llvm's linker) is the fastest linker in town by far.

Interesting. I searched for some more information on that but don't see anything except the GitHub repostoryitself. I wonder how would one go about integrating that into .Net stack.

Check out Bazel's remote execution feature and Google RBE (remote build execution) [1]. Bazel itself is language agnostic (language support comes in the form of rules).

(disclaimer: I'm an engineer on the Bazel team)

[1] https://docs.bazel.build/versions/master/remote-execution.ht...

There's also an opensource goma client + server which proxies to RBE (or potentially other bazel remote execution API implementations). Which is great if you can't or don't want to use the bazel client, but can use a compiler wrapper.

https://chromium.googlesource.com/infra/goma/client/ https://chromium.googlesource.com/infra/goma/server/

Nice info. I've wondered for quite a while about the impact of the G level of compile parallelization :). A couple of followup questions, if I may:

- How long do incremental and non-incremental builds take with the build configuration you describe?

- How does core<->engineer allocation work? Is 500 cores an average number? High-priority work presumably commands more access, but I'm curious if allocation is generally team-specific or engineer-specific (eg, more responsibilities = more resources). (I guess the reason I ask this is that 500 cores sounds kind of impressive and I'm going "surely they'll run out of cores if they're handing them out like candy???"), but then thinking about it that's only like 11 computers...)

- I'm curious if you happen to know how Chromium gets built (IIUC Android builds it 2 (or is it 3?) times). It's tricky to distill (presumably/understandably deliberately so) how many cores the CI/trybot/etc infra is using. Overall my curiosity is what the core/time ratios/graphs are, ie how many cores translates to how short of a build.

- As an aside, I vaguely recall absorbing by osmosis several years ago that Chromium takes approximately 15 minutes to build on a Very Cool™ local workstation (HP Z series). Not sure how out of date this info is. Do engineers still do test builds on local hardware (and if yes, how long does this take? probably longer than 500 cores, heh); or is everything done in throwaway cloud instances nowadays?

Thanks in advance for whatever answers you're happy to provide :)

Google has unlimited resources. Cores and codebase size don't matter to them. Only concern is what's on the critical path, e.g. final link step. It's very rare for builds to not be incremental. AFAIK JavaScript is the only language that can't be compiled incrementally. Probably the only engineers who would need to run a build solely on local hardware are ones that help support open source tools.

Js can and does get compiled incrementally if people do it correctly. It's fully supported by blaze. It's just that a bunch of old codebases globbed everything, so it appeared no incremental.

Is that with the js_binary/library rules or something new? Last time I checked, all the work that goes into compiling the JS sources and minifying them into a single "binary" blob happens in the js_binary definition, and the js_library definitions traditionally served only to construct a total ordering of raw source source file dependencies.

Two partial answers: there's a newer better thing thats now the default and splits the building, but even under old js_library, there were some gains from not globbing everything together...

Distcc is neat, and invaluable, but it's parallelism on a different level to this, which is introducing parallelism at the single-file compilation level. It would be interesting to see the overall performance impact of this in combination with parallelism on a build level. It could even be harmful (but I doubt it) due to resource contention.

That's great for super large builds, but the point of this is to deal with "I have one file I want to compile which is taking 30 seconds to compile, and I only have like three files to compile, so it would be awesome if I could use all of the cores on my laptop to do this compile and get it down to 10 seconds or something".

This, parti ularly for template heavy C++.

Older versions of Stan on GCC 4.8 would take 60+ seconds to compile a single C++ file.

I write template heavy code. I’ve recently refactored template instantiation across many object files. It’s faster, but a lot of work. It’d be great if I didn’t have to.

distcc can be useful, but requires that the local build environment matches the remote 100%.

No it doesn't, it sends preprocessed files just to not depend on remote env.

> it sends preprocessed files just to not depend on remote env.

to then be compiled by the host compiler ..

Sure, one compiler can't be egcs 2.95 and the other clang 8 (at least not in most cases) but the target compiler system doesn't even have to have include files, libs, libfoo-devel packages installed or they can be, but from old versions. As a proof of concept I once had a linux with gcc4.2.1 make c->.o files that later ran on OpenBSD. Would I trust that resulting binary? Sure not.

But in cases of "lets spin up 10 compile boxes with <same ubuntu within a week of updating>" or have something along the lines of Xcode which allowed you and your colleagues to help out each other with small work units in order to make a single compile run quite much faster, that is a definite possibility.

For that final release build, you might consider running it on one of those 10 VMs in the example above and have 'only' 9 others help out with the sub-parts in order to get some kind of .. guarantee.

If it takes a minute for a full build on one box, getting a hint after 10 seconds that you misspelled something and that it won't ever link decently is worth something too if you value your time as a developer.

Not saying it's perfect, only chipped in on the "must be 100% or it can't ever work", hopefully without needing to go into the "coach says we need to give 110% this game, and 120% if it's the finals" nitpicking.

This doesn't work for C++, as there is no standard ABI and thus no safe way to link object files from different compiler versions into the same binary.

Icecc (an old distcc fork) also distributes the toolchain, to avoid that particular problem.

https://github.com/icecc/icecream

Reply on news.ycombinator.com