Tup – an instrumenting file-based build system | Hacker News Reader

Tup – an instrumenting file-based build system

gittup.org

92 pointspabs336 comments

I spent a lot of time exploring make replacements a while back. I really like Tup as a "update outputs based on inputs" system. A lot of people get hungup on it though because there are things that make can do that Tup can't, as Tup is much more of a special-purpose tool than Make.

"make install" for example is specifically something Tup can't do.

This is trivially solved by having "install.sh"

There are also various limitations, caveats, &c. such that I typically had a "build.sh" or some-such that did any necessary pre-tup work.

The most restrictive rule is the set of output pathnames cannot depend on the contents of input files; they must be fully realizable from just the set of input pathnames. Calculating this outside of tup is fine even if it's expensive, as long as it changes rarely. Tup will error if the list is wrong, so you can't accidentally build with out-of-date outputs.

> "make install" for example is specifically something Tup can't do. This is trivially solved by having "install.sh".

`make` is different things to different people. What you describe, `make` as a tool to compute optimal strategies for recursive file-based build operations, is indeed its original purpose.

On the other hand, I tend to use `make` more as a standardized format for script collections. Whenever I open a new repository, and it has a Makefile in it, I'm very thankful because it's a useful jumping-off point for how to build and use the software contained in the repository. Therefore I do the same in my own projects. Even if the project is in a single language with established conventions, I tend to include a Makefile regardless as a way of documenting.

For example, if I have a Go project, I could rely on other people looking into it to know how Go is compiled, or I can just put in a little Makefile like:

  PREFIX = /usr
  
  build/foo:
    go build -ldflags '-s -w' -o $@ ./cmd/foo
  
  install: build/foo
    install -D -m 0755 build/foo "$(DESTDIR)$(PREFIX)/bin/foo"
  
  .PHONY: build/foo install

Which will make it entirely obvious to everyone, including non-Go-developers, how to build and install the software.

See my comment to a sibling post, but I originally had something just like that with "tup" rather than "go build" but people complained so I switched it to "build.sh" and "install.sh" and people stopped complaining.

> This is trivially solved by having "install.sh"

> There are also various limitations, caveats, &c. such that I typically had a "build.sh" or some-such that did any necessary pre-tup work.

Frankly if I need to write custom wrapper scripts for a build system then the build system isn't up to scratch. On every project I work on, you can run

    make list

To get a list of targets, and build/install/predeps/deploy are all defined. On any of of our projects, make predeps && make build will output a deployable binary.

> Calculating this outside of tup is fine even if it's expensive, as long as it changes rarely.

So no include files? That's another deal-breaker!

Include files are fine; in fact tup automatically handles that for static headers (i.e. header files that aren't generated as part of the build process).

One example of what I'm talking about is what another commentor mentioned: .java files will generate separate .class files for each inner-class in the file, so given "foo.java" you can't tell which .class files will be output when compiling "foo.java" without looking at the contents of "foo.java"

I actually originally had a Makefile with predeps/build/install targets where the "build" step just called tup. People complained that "if tup is supposed to replace make, why are you using make?" I switched it to predeps.sh/build.sh/install.sh and everybody shut up. I personally think it's less ergonomic but don't care enough to argue with a different person every week.

> I personally think it's less ergonomic but don't care enough to argue with a different person every week.

I'm with you here!

> So no include files?

Huh? This definitely does work with Tup as expected.

> So no include files? That's another deal-breaker!

Tup monitors any files that are read while the build command is running, and implicitly adds them as dependencies.

Build systems shouldn’t be installing things anyway.

On one project that supports both tup and cmake: building from scratch with tup takes 8 seconds, ninja takes 18s and "make -j" takes 22s.

If you are a heavy shell user you'll also appreciate that tup can be executed from any subdir within the project, no need to keep around extra terminal just for compiling.

That means that the upper bound on doing nothing but compiling the stuff is 8s. to which make adds 14s of pure waste. That's a weird situation which seems like having a low probability of being broadly representative.

Yeah, meaningful benchmarks take a lot of work.

- Does the reported ninja time include the cmake step that generates the ninja files? (which typically doesn't need to be done for incremental builds).

- "make -j" doesn't limit the number of jobs, so you can run into resource contention if building too many things at once. Ninja (by default) limits the parallel jobs to the number of CPUs.

- Disabling make's built-in rules (with `-r -R`) can make a big difference to performance.

- Make spends a lot of time processing the ".d" files with header dependency information -- 98% of the time in my (artificial) benchmarks!

There shouldn't be any .d files in a from-scratch clean build. They get generated during compilation and are not needed initially because every object file is out of date due to its nonexistence.

However, they do add a lot of bulk once made and increase with the size of the project.

A lot of the information is redundant. Among the files you may find many pairs which only differ in the principal file, the headers being the same:

   foo.o: foo.c $SOME_HEADERS

   bar.o: bar.c $SOME_HEADERS

In theory, if we have GNU Make, we should be able to compress these into static pattern rules. Concretely, in the above case, the pair condenses to the static pattern rule:

   foo.o bar.o: %.o: %.c $SOME_HEADERS

When, we check whether, say, bar.o is out of date, we find that it as a target in the left side of the rule. Then in the middle, the name matches the %.o target pattern, identifying the stem as "bar", and from there the full dependency rule is instantiated with all the prerequisites.

To generalize that, we partition the $(OBJECTS) into subsets based on which patterns they belong to. Hopefully there are a lot fewer subsets than files.

Then for each subset we do

  # The a.h b.h subset
  $(OBJS_0): %.o: %.c a.h b.h

  # The a.h d.h subset
  $(OBJS_1): %.o: %.c a.h d.h

and so on. The only problem is managing these subsets (in particular, incrementally) in such a way that more time isn't spent doing that than time saved loading the rules.

Stale dependencies can be troublesome when header files are renamed or deleted; this sort of scheme could make it worse.

One way to handle these subsets incrementally would be to keep them in some separate database outside of the Makefile. Whenever a file is compiled and the raw .d is generated, we run a script which looks up the header signature and matches it to a subset. Then it assigns that file to the correct subset (perhaps deleting it from an existing one). A step at the start of the build would regenerate the dependency file seen by make if the database indicates that it's out of date.

Ninja stores the header dependencies in a database and then deletes the “.d” file. As far as I know it doesn’t do your deduplication – presumably because it hasn’t been necessary; reading and parsing hundreds of “.d” files was the slow part.

This wasn't a "meaningful benchmark", just a single data point to show how tup makes practical sense, when everyone else was criticizing "yet another build system" on theoretical points.

I included the time to generate ninja files, the measured time is for project to build from scratch. Tup generates its initial files _much_ faster (~0.2 seconds compared to ~4 seconds). The strength of tup is actually in incremental builds, but that's harder to measure: you get shorter times with larger relative variations between repeated tries.

I wasn't interested in working around the bad make defaults, I also didn't spend any time optimizing tup flags.

I suppose these numbers must converge for all three as number of files go up? (And compile time for each file is not negligible.)

I say this because in my use case ninja spends almost all of its time running compiler processes, not resolving deps.

I did hit a weird edge case in Make's job scheduling. Ninja will try to build things in the order that they are listed in the "build.ninja" file; make kind-of does too, but if any dependencies need to be built first, the later targets get pushed to the very end of the job queue. This means that you can end up running many resource-intensive linking jobs all at the same time at the very end of the build: https://lists.gnu.org/archive/html/help-make/2016-11/msg0000...

It's hard to say how much impact this would have in real-world scenarios.

I think I must have hit that case often, I have come to see it as inevitable that Make does a lot of linking at the end...

I used to use this _extensively_ years ago. It sounds great in theory but very quickly gets in your way.

A while back I went through several of these kinds of build systems. The one I eventually settled on was Snakemake[1], with which I finally was able to put together an entire complex bioinformatics workflow[2]. I would say that Snakemake's strength is one of Perl's old mottos: "making easy things easy and hard things possible." On the easy side, writing a simple rule with fixed inputs and outputs is just as easy as Make (trading the weird whitespace rules of Make for the less weird whitespace rules of Python). On the hard side, it supports stuff like computing inputs at runtime, or using regex capture groups to determine output file names from input file names. So for example you can have your rule for LaTeX to PDF conversion read the source TeX file, determine all the external images, bib files, etc. referenced from it, and add those files as input dependencies so that when any of them changes, Snakemake will know to rebuild the PDF. See e.g. this rule[3] that builds my resume PDF from the LyX source.

[1]: https://snakemake.readthedocs.io/en/stable/

[2]: https://github.com/DarwinAwardWinner/CD4-csaw

[3]: https://github.com/DarwinAwardWinner/resume/blob/main/Snakef...

I really tried my best to use Snakemake in ~2018-2019 for day-to-day data science work. I found that its DSL was a confusing and (at least at the time) under-documented hybrid of Python and custom syntax. It also suffered from some of of the primary limitations of traditional Make:

• Targets/outputs must be files, without even the opportunity to extend the space of possible targets e.g. using your own hash comparison function. This is not suitable for a workflow where the output is a directory of filenames that aren't known in advance, or a database table.

• By default, can only run commands through a shell, not directly as a subprocess. At the same time, it also lacks support for nontrivial chaining and composition of commands. In your CD4 example you hack around this by literally writing your own pipelining command runner in Python; hard to say if that's any better than a big blob of shell script in a string!

I agree that it definitely "makes hard things possible", but my experience did not give me the impression that it "makes easy things easy". It was frustratingly deficient in what I believe are obvious areas for improvement, and some of the most obnoxious Make pain points that I encounter on a regular basis, even for regular day-to-day software development.

I ended up switching from Snakemake to DVC basically as soon as it came out, because it at least supports using directories as outputs, and has a few other nice features that specifically work well for data science project version control and artifact management/sharing for collaboration. That said, I had/have much less of a need for complicated workflow orchestration than for easy tracking & sharing of datasets and large artifacts (fitted models, processed tabular datasets), and for tracking directories as outputs (partitioned Parquet datasets).

It would be interesting to go back and re-evaluate the space of workflow DSLs and task runners, because DVC does not have the complete feature set that Snakemake has, and IMO nor should it (I am already getting worried about its feature creep). I'm also not sure if or how well Snakemake and DVC could be made to work together; the former for task running and the latter for task input/output tracking with version control. The perfect tool (or better yet, the perfect combination of tools) remains elusive.

I very much agree with you about DVC's feature creep. The other issue I have with it is speed. DVC has left me scratching my head at its sluggishness many times. Because of these factors, I've been working on an alternative that focuses on simplicity and speed[0]. My tool is often five to ten times faster than DVC[1]. I'd love to hear what you think.

[0]: https://github.com/kevin-hanselman/dud

[1]: https://kevin-hanselman.github.io/dud/benchmarks/

Thanks! I really like your clear explanation of how Dud differs from DVC (and I prefer your version in all cases).

Would it be possible for Dud to push/pull from a DVC remote and use the DVC shared cache? That would be really useful so I (iconoclastic free software user) could use Dud on my machine/acocunt, but still share data and artifacts with other people (who don't give a shit what tool they use) using DVC on their machines/accounts.

Also: Does Dud support reflinks at all? Or does it only support symlinks?

Unfortunately, there's a few things that currently hinder compatibility with DVC caches. First, Dud uses the Blake3 checksum algorithm, and DVC uses md5. This means the content-addressed caches will have completely different file names. Second, directories are committed to DVC differently than they are in Dud. For directories, not only will the committed file names not match (due to point 1), but the contents will not match either. Both of these things could be addressed, but it would take a lot of effort and would likely cost Dud in terms of its two main goals, speed and simplicity. I'm not opposed to this if we can make it work, though.

Dud currently does not support reflinks, but I think adding reflink support would be fairly straight-forward. Just curious: What filesystem and OS are you using for reflinks?

I'd be happy to chat more about this. Feel free to open GitHub issues for these items. I welcome contributions as well. ;)

Hehe, I forgot about the pipeline function. I had originally written that for something else and then repurposed it here. It always felt like something that ought to be built in to python.

And I guess I'm kind of grading "easy" on a curve here, because the first build tool I tried for bioinformatics was GNU Make, and it was not up to the task, even when I exploited all the special GNU features to the hilt.

Why was it chosen and why did you regret that choice? Could you explain a bit more?

I wanted something faster than Make that was cross platform, that let me also define my own processes instead of needing high level "language aware" build systems.

Tup fit nicely, until build variants or Java were needed.

Aside from not really working well on Windows, at least at the time, variants were completely broken on Windows.

Further, compiling any Java with inner classes was impossible since they emitted extra class files for each inner class, which meant the build artifact hierarchy depended on the code - something Tup isn't prepared to handle.

It just got really annoying over time.

Ninja-based systems quickly replaced it for me.

I wonder if tup devs are aware and interested in adapting their model to solve the multiple byproduct per file.

Last time I dabbled with tup, I remember it was not trivial to support any process that generated more than one output. My use case back then was lex / yacc: you can generate a source and a header file with them (and even some other output files), but it was not trivial to capture these dependencies with tup.

I’ve been using it for personal C++ projects for a few years (because I despise CMake) and find it quite pleasant. You do have to do things it’s way, though. Don’t expect it do adapt to your structure, you may need to rethink things slightly. Also don’t expect it to be a one stop build tool, I still use a simple makefile and “make init” to build dependencies that need to be compiled with something other than Tup, and then use Tup only to build my own sources.

I migrated my main personal project (a programming language written in C) to Tup recently. So far I love it, it seems to 'just work'. Some highlights:

No need to map header (or other) files to C files. Tup detects when files are read and adds them as implicit dependencies.

Environment variables (that are used) are considered build inputs, so no need to `make clean` when you need a build with debug symbols.

---

Admittedly my project is quite straightforward, but I thoroughly recommend giving Tup a try.

KolibriOS[1][2] uses it as a primary build system.

[1] https://kolibrios.org

[2] http://websvn.kolibrios.org/listing.php?repname=Kolibri+OS

See also gittup, an experimental linux distro where the entire linux kernel + userspace is built using tup. https://gittup.org/gittup/

Tup has some automated performance tests, which I co-opted to benchmark Ninja vs. Make: https://david.rothlis.net/ninja-benchmark/

nice, my biggest problem with c is macros and how to build something. Make files are freaking horrible, tup seems like a nice alternative that keeps it simple. Simplicity is important. Complexity is the result of bad design.

is it like nix?

Reply on news.ycombinator.com