Hacker News Reader

Comment by Alecazam | Hacker News Reader

I feel like make/ninja suffer from the same output modstamp > input modstamps to rebuild. Really a build system should track modstmaps on all inputs (apps included), and the rebuild on not-equal.

Would be nice to see build2 go this route.

View on HN

I did some exploration of this idea in a followup build system! See https://neugierig.org/software/blog/2022/03/n2.html . (It's not really production-ready.)

Why not just embrace Content Based Addressing and the Bazel action cache? Not necessarily adopt the RBE protos, but you could have on-disk cache that are similar or even identical.

Nowadays, I think a build tool that doesn't natively support distributed caching (and possibly remote execution) is a weird choice. And I don't think that spawning layers of processes allows for good parallelization as you don't know if an action is going to be network bound or compute bound and the job slot is then spent. So you either oversubscribe or undersubscribe.

Sadly, I think things like distcc or ccache have mostly gone out of mode. Vertically scaling (multi core, multi thread, multi TB RAM build machines) and simple artifact caching have won for now. Artifact caching with dependency tracking in the midst of job scheduling was solved “good enough” by OS based mechanisms. Having used and managed the basics of a build farm, removing the need for networking, build farm management and build job coordination is a huge burden relief when your source tree doesn’t require it.

Part of the motivation to use processes is because their structure helps to keep the job generic, uni

Reading about manifests immediately reminded be of Clearcase/Clearmake. My memory may be not correct after all the years passed, but configuration records for derived objects contained the same information as proposed manifests + some more metadata about source (versions, at least). If I remember correctly SCons allows to implement a custom "checker" (or whatever it is called there) to compare files based on any criteria that are useful, not just timestamps. But this is about the only visible advantage of SCons over Make, at least for our case.

I stumbled upon the very experimental nix-ninja [0] recently.

They seem to create a dynamic nix derivation per compilation unit, which would be very similar to what you describe as manifests in your post as it also creates a hash of all inputs.

Would be interesting to here your opinion on that approach

[0]: https://github.com/pdtpartners/nix-ninja

I really like the ideas you have in n2. Hashing the entire build string is a great idea. And it really is as simple as just having all the timestamps of the inputs in the hash. And you have the output timestamp as a quick test.

Reply on news.ycombinator.com