Hacker News Reader

Comment by saurik | Hacker News Reader

Meanwhile, I am deeply concerned about modules, as it seems like the kind of feature that is going to massively harm build times by making it much more difficult to do truly independent distributed builds (which right now is pretty easy with C/C++ due to its separate compilation and extremely explicit inter-file dependencies).

View on HN

Modules make separate compilation possible. C++ does not really have separate compilation right now.

I do not understand this comment, as I have done separate compilation every day with C++ for over two two decades: the model inherently only allows this, as every translation unit absolutely must be compiled separately and then linked... you don't even have a choice in the matter. I have also used both off-the-shelf and custom distributed build systems (most recently, I have deployed one which does insanely-parallel builds of a C++2a codebase using AWS Lambda), which pair extremely well with the explicit file-oriented dependencies that can be reported by each translation unit and used to determine which parts of the build are now stale given changes to the filesystem. I have never seen languages with module systems support even basic separate compilation, and it terrifies me; meanwhile, the flags I have seen being added to clang for attempting to preserve separate compilation behavior with the modules spec doesn't leave me feeling good about the endeavor. It could be my fears are unjustified, but it is extremely strange and somewhat annoying that the advocates for it don't ever seem to appreciate the state of the art for the status quo.

> I do not understand this comment, as I have done separate compilation every day with C++ for over two two decades: the model inherently only allows this, as every translation unit absolutely must be compiled separately and then linked... you don't even have a choice in the matter.

Except if your headers include macros or templates, which must be evaluated every time. This is not separate compilation. What C/C++ developers have been doing for 20 years is jumping through hoops to incrementally reuse previously compiled results because C/C++ violate basic modularity principles [1]. They call it "separate compilation", but it's a pale shadow of true separate compilation.

Proper modules would enforce abstraction boundaries so true separate compilation becomes possible. I can't speak to the specific C++ proposal here, but enabling separate compilation is exactly what modules are for.

[1] http://llvm.org/devmtg/2012-11/Gregor-Modules.pdf

Normally use of templates do break separate compilation but it can be recovered, and it is routinely done on large codebases, via explicit template instantiation. The ergonomics are not great of course and hopefully modules will improve things.

With C++, you have a lot of limitations on cross-machine, cross-library compilation. The library is not defined just by the platform, but by the compilation directives and macros.

Since headers can redefine macros, that also means that even including headers in a different order can change the meaning.

Older languages like C++ with headers are simply source files with a different extension, so your individual source file compilation is actually loading in and parsing cross-sectional subsets of your full project. Modern languages typically define the interface inline with the code, and export that interface to other modules which would consume it. The reason that you don't have intra-module distributed build is because the intra-module code dependencies are implicit - there is no developer-defined partial cross section of the code, all of the code needs to be pulled in and semantically evaluated as a unit.

You could hypothetically still do distributed optimization of the code after it is semantically evaluated, but TBH there is a lot of time wasted in evaluating headers for each file. Modern compiled languages usually have compilation time as a restriction on the implementation and design, and elimination of the need to repeatedly parse and semantically evaluate files is part of their design.

> The reason that you don't have intra-module distributed build is because the intra-module code dependencies are implicit - there is no developer-defined partial cross section of the code, all of the code needs to be pulled in and semantically evaluated as a unit.

And this seriously somehow doesn't seem like a problem to you? My massively distributed (to AWS Lambda) C++ build compiles at the speed of the slowest single translation unit. You seem to be optimizing for serial compile performance, but if you truly want speed you have to go parallel. These explicitly-managed "cross sectional sunsets" are what makes this so easy to do with C++.

There are definite tradeoffs here, but one thing to consider is that dev tooling generally can't benefit from that kind of massively distributed parallelism. If you make a change to a widely-included header file, code intelligence tools suddenly have to do a huge amount of work to reevaluate their internal model of all affected translation units. This can really result in an unpleasant and laggy editing experience, even on medium-sized projects. That's the kind of thing that modules can help with; I certainly wished fervently that they were already a thing when I used to work on C++ static analysis tools years ago.

You can still do distributed builds with modules. You can't have cyclic dependencies anymore though. Your code has to be structured into logical modules that are meant to be built in parallel. It's not going to be free. You could have replicated modules with the old preprocessor style, but this way it is formalized and enforced by the compiler.

Wasn't the point of modules to decrease compilation/build times?

Some claim this, but I haven't seen it play out this way yet in what I have seen for modules; it could be that people are hyper-optimizing for serial performance when the future (and present!) is parallel? If so, :(.

This article is possibly out of date--or maybe was wrong to begin with--but it shows how the modules concept from earlier this year simply broke parallel build efforts by enforcing a ton of serial dependencies.

https://vector-of-bool.github.io/2019/01/27/modules-doa.html

I have also seen this. My guess is that it highlights more that there are way too many dependencies in the libraries being used and that those dependencies aren't organized well.

Reply on news.ycombinator.com