Hacker News Reader

Comment by nneonneo | Hacker News Reader

The only relevant info I could dig up was this issue from Kubernetes in 2016: https://github.com/kubernetes/kubernetes/issues/27444

It seems to be discussing the speed of building Kubernetes on their build-servers, targeting multiple architectures at once. Unfortunately all the build servers are internal to Google so I don't know exactly what's being included (or what's being run), but this comment (https://github.com/kubernetes/kubernetes/issues/27444#issuec...) details the breakdown of time spent:

    30m of build
    7m on cluster startup
    12m on tests
    7m on cluster teardown

So, taken with an enormous hunk of rock salt, it seems like Kubernetes might take around 30m to build on a cluster, to build (possibly multiple) release artifacts. Seems in line with Rust.

View on HN

That's compiling for 10 os/arch combos, so ~3m per.

That makes a lot more sense. Go was specifically designed to compile fast.

On the other hand, these are running on some big cluster with a whole bunch of CPUs, so I wonder if you’d still get 3m per build on something like a laptop?

Any individual build they're referencing there runs on a single vm (the cluster referenced there is the k8s cluster being built up and torn down on that vm to run e2e tests on). It looks like at the time they were using GCP standard-8 instances, which are 4 core/8 thread w/30GB of ram. It's hard to say exactly what the current build time is, as it looks like they do some build caching now; on my desktop a fully clean build of current kubernetes master (3.2 million loc) for linux/amd64 takes just under 2 minutes.

What’s the state of parallel builds? We’re getting 16/32 core cpus. In a few years, we’re all going have Threadrippers.

Of course everything can’t be done in parallel.

Almost all parts of the compiler (at least for LLVM) are still strictly single-threaded. You can't even use threads yourself if you write your own LLVM module pass.

Some of the parts of a compiler are inherently serial. e.g. parsing a file. But if you have to parse multiple files, then hey presto, parallelism. The linker has to work as a spooler for writing the resulting executable or shared objects, but afaik, there's no reason for symbol resolution to not be done in parallel. And if you want to optimize your GOT then that could also be sorted by a page rank style optimization which is the algorithm which was the foundation of the phrase 'embarrassingly parallel'.

Given that most sweet shop ITs provide i7 and i5 with 8GB and HDD, good luck getting that hardware adopted there.

Reply on news.ycombinator.com