Hacker News Reader

Comment by peterkelly | Hacker News Reader

One has to step back and ask: "What made you decide to introduce non-determinism into your compilation process in the first place?"

View on HN

Nobody sat down and said "yes, I'll make this nondeterministic". If you look at the wiki page's sampling of different issues -- https://wiki.debian.org/ReproducibleBuilds/Howto#Identified_... -- you'll see it's a mix of various things:

* output is deterministic but dependent on some aspect of the build environment (locale, hostname, etc)

* output is accidentally non-deterministic (eg "we put all the .html files into a tarball with a shell glob pattern, which gets you an order dependent on your filesystem implementation and the phase of the moon")

* a wide array of "output contains a timestamp" issues

"Let's put the timestamp into our version string/a generated file so the user knows when it was built" is a really common thing, and it seemed like purely a nice convenience feature until the concept of 100% binary-reproducible builds became a current concern.

Curious: What's the usual fix for this class of problem? Is it possible to flag certain bits of data as known-to-change and evaluate the rest of the build in isolation?

Example: Say you have a version string that outputs a build time. Can you hash the program with just that bit of string data marked as unknown (or that string table entry replaced with a placeholder, as far as our verifier is concerned) and verify the rest of the program is unchanged across builds?

It varies by case.

For the particular example of time, some common approaches are to hook the sys call for time such that it returns a constant value, or to teach your tool to output a constant value.

The Nix thesis (https://nixos.org/~eelco/pubs/phd-thesis.pdf) is a great read and goes through a bunch of these examples as part of its research.

Typically you just stop including the build time. It's a deterministic build, so why would the build time matter?

I was going to say "because it's convenient", but I realized that you're right: if it's identical anyway, the release date is as good a timestamp of the code as you're going to get. If for some sysadmin reason the build time is relevant, file metadata can tell you.

Maybe I want to know that someone was messing with stuff, and even if they put something 'back in place' that matches, I may still want to know that it was built this morning and that something is going on.

Of course code signing is going to mess that up.

This only works in some limited circumstances; more often, the build differences cascade and produce a completely different result.

For example, if the compiler puts full path names in intermediate object files, so on a different computer the object files have a different size, this may result in choosing a completely different layout for the binary.

You leave it out and go off revision hash or something. Back when we first introduced build times it was pre-modern-source-control.

The Rev hash fulfils the function now.

https://reproducible-builds.org/docs/ has a lot of strategies. For example, you could use the SOURCE_DATE_EPOCH environment variable, which if set, uses that value instead of the current actual time. This means that your normal builds do the normal thing, but you can configure the variable and get a reproducible build.

> Is it possible to flag certain bits of data as known-to-change and evaluate the rest of the build in isolation?

This is what I do. I'm working with a hardware vendor that is not particularly security-conscious. I have to send them compiled binaries as part of a build pipeline, which they then modify in a non-deterministic way. Fortunately the changes are limited to data which has no impact on the function of the software. So I've created hashes of all the individual components that make up the binaries, and I skip hashing the parts that they modify.

It's not ideal, but I can be reasonably confident that the binaries I compile function the same as the binaries that my hardware vendor distributes.

You fix the tools so that they work properly. You modify the compiler to always produce deterministic output (e.g. dummy timestamps).

Except for the first one, the last three are great examples of why I don't care about 100% binary reproducible builds as an end user.

Knowing when something was built is more valuable to me than a reproducible build. Seems like a fad triumphing over common sense.

Users often don't care about things until they become a problem. For example, most users don't care how their password is stored on a server... until it gets compromised. Also, timestamps in output usually indicate source code age, not build date. I have a package with '2015' in the version but it was built in 2018.

That's a condescending non-answer.

We're not talking about passwords in a database, we're talking about reproducible builds. I already trust Debian, so it's irrelevant to me, as a user, if their builds are 100% reproducible. Can you explain why I should feel differently?

Although I don't think it's a valuable thing to spend time on, it's not my time, so I honestly don't care too much.

From an end user point of view, probably the most important thing it allows Debian developers to do is verify that the binary packages they are shipping are the packages they intended to ship. You trust them, they have to trust their tools... so they're making sure their tools tell them about yet another situation when something might have gone wrong. (it could be something innocent or malicious, but without the tooling, they might not even be aware of the issue)

It doesn't: there can still be backdoors or compiler errors before the hash of the binary. That's why DO-178C certification requires proof of source-to-binary equivalence. It's also why CompCert and CakeML were built. Stopping backdoors requires doing that with independent review for requirements, design specs, security policy, and implementation with them all corresponding.

People concerned about this stuff are putting the least effort into what causes the most failures. Reproducible builds are a distraction if what's reproducible is 0-day-ridden software whose safety/security checks might have been removed by buggy, black-box compiler.

You are right that this isn't the be-all, end-all but is merely one of the pieces needed. I was trying to give a layperson's reason for caring (i.e. what was being asked) which often involves oversimplifying things.

Oh, you explaining it is a nice thing to do. I was just showing what was actually necessary to meet the intended goal since that's my field. I learned a lot of the problems from people like Myers who wrote the book on subversion back in 1980. Current approaches still aren't as thorough as what they were doing in the 1970's-1980's.

https://csrc.nist.gov/csrc/media/publications/conference-pap...

If it's about repos, I recommend David A. Wheeler's page on SCM security since following it eliminates most of the risks.

https://dwheeler.com/essays/scm-security.html

If the build is reproducible, why does it matter when it was built?

Knowing the build was reproducible tells me nothing of value.

Knowing the timestamp of when it was built, or the source was downloaded tells me roughly how old the package is, which can be useful for a variety of reasons.

Do you care when the build was made or from when the code build into it is? The former is a problem for reproducible builds, the latter is possible.

I can do a build off of year old code.

Would you rather you get a timestamp embedded that tells you it was built today, or take the hash and be able to find the exact revision.

>Knowing the build was reproducible tells me nothing of value.

It does. It proves that the given package was produced by the given source code. However, you can keep the timestamp as well! One of the goals of reproducible builds is to record values like this so it can be replayed later at a future point in time. This is done by the BUILDINFO format.

https://reproducible-builds.org/docs/recording/

You don't care about the build date, you care about the release date, which is specifically what debian is trying to achieve with SOURCE_DATE_EPOCH.

> Knowing the build was reproducible tells me nothing of value. It means you can uniquely address what you wanted by what you got (and vice versa). This makes storing and retrieving things much easier, scalable and secure. It also makes it much easier to ensure that you got what you wanted, and someone else did too, on their machine.

Reproducible builds allows for better auditing which makes it more difficult for malicious maintainers to plug backdoors into systems.

What's the relevance of knowing when something was built if you can check the source version and rebuild it yourself in the exact same way? If the build is reproducible it's merely a cache. You could potentially recompile it every time you try to execute the command and you'd get the exact same thing.

Knowing when something was built is more useful than having better verification of exactly what code went in?

Besides, it's a deterministic build. You can use a lookup table with the hash to find out exactly what revision it is.

You don't really 'introduce non-determinism'.

That's like saying, "why did you introduce entropy?" It's there. It happens, and it's difficult to get rid of in any complex system.

https://en.wikipedia.org/wiki/Reproducible_builds

https://wiki.debian.org/ReproducibleBuilds

https://www.win.tue.nl/~aeb/linux/hh/thompson/trust.html

Debian was one of the first large distributions to use a build farm for many different architectures. Maintaining deterministic builds (even as far as just file timestamps) in a distributed environment is very challenging.

If you aren't thinking about reproducibility things like putting timestamps in the binary seem reasonable. Which is what lots of tools did. Hence the problem.

This is true and it's also the model I use to avoid bugs. Any time I'd add a bug to the codebase, I just look at it and decide not to write the bug. In this way, no bugs are ever introduced.

Reply on news.ycombinator.com