Hacker News Reader

Hacker News

Comment by Godel_unicode | Hacker News Reader

Parent Full thread

Godel_unicode

There are plenty of situations where the output of the build process is not deterministic on only the source.

View on HN

Nix hashes your development environment along with the sources to generate a version hash of the build process result.

Example?

Install Python from source with and without readline-devel in the ld library path.

Or any other piece of C software with macros that depend on the build environment. Same .c file in, different binary out.

Nix does not have libraries in the dynamic linking library path, they are made available for linking by specifying the library's derivation as a dependency. There is no such thing as e.g. a global glibc or readline. E.g.:

  % ldd $(which nc) | grep ssl                                                
 libtls.so.18 => /nix/store/s6j0yd68cnfb4mv76lyrb413qhhac57g-libressl-2.8.3/lib/libtls.so.18 (0x00007f8875f6d000)
 libssl.so.46 => /nix/store/s6j0yd68cnfb4mv76lyrb413qhhac57g-libressl-2.8.3/lib/libssl.so.46 (0x00007f8875f1a000)
 libcrypto.so.44 => /nix/store/s6j0yd68cnfb4mv76lyrb413qhhac57g-libressl-2.8.3/lib/libcrypto.so.44 (0x00007f8875d3b000)

A binary or library always links against a specific version of a library. This is why in Nix, you can have several versions of a library installed in parallel. The initial part of the path after /nix/store is the hash of the derivation used to build this version of libressl and all of its recursive dependencies. Furthermore, Nix can build packages in a sandbox (which is the default on NixOS) to avoid accidental dependencies.

Consequently, a Python derivation that depends on readline would have a different hash than one that does not depend on readline.

Homebrew has had to deal with this in its sometimes-painful transition over the years from being a build-from-source system to a binary package manager. Suddenly you can't just have some package require the boost package be built using the --with-python flag, you either need to have it on in the bottled version (but wait, then boost has a dependency on Python, is that okay?), or you have to ship a separate boost-python, like Debian (but wait, then it's no longer a 1:1 mapping of recipes to bottles, is that okay?).

Nix handles this just fine. You should check it out.

This sort of thing is IMO exactly why containers have merit. Let the software vendor decide what to distribute or others jump in and fill the gap. Powerusers can still read the dockerfiles and decide what do do or even build themselves. When it's built the image is well defined, i.e. not depending on host environment, including files stem except volume mounts. When will we get a whole desktop distro like this?

Many of the pieces which go into a desktop have a strong assumption of being able to interact with other components in ways other than a network socket. Containerizing an application is one thing, but how do you containerize something that has plugins? What about desktop widgets? Do they go in their own containers, or get mixed in with the host's somehow? How is ABI compatibility enforced now that we're in container land and none of that matters any more?

What about client/server components where there's dbus in between, like the network manager or volume control?

These are solvable problems, but it's been enough of a challenge building a server OS from containers (RancherOS), and building a desktop OS that way is a significantly harder problem.

> containerize something that has plugins

I don't really see the problem there, I see the plugins as essentially just data (i.e. stored in a mounted volume), for which updating and versioning is in the domain of the application itself or maybe some standardized library it uses.

desktop widgets: essentially the same thing, it's a plugin to the desktop environment and can be stored as a volume mount on the DE container.

dbus is probably something that would require an evolution on the container side, or alternatively it would need to be all abstracted into network interfaces. another possible way to look at it is to have a layer between kernel and containerized userland that is responsible for manipulating all the physical host things in the traditional way, and the examples you give are exactly that. maybe this sort of thing should continue to be distributed tightly together with the kernel.

There's an entire Debian project working on making Debian builds deterministic and/or reproducible. The fact that they had to devote a whole team to it and they've poured who knows how many hours into it kind of says that most software builds aren't reproducible (deterministic, theoretically, yes, cause in many cases it's just different timestamps).

NixOS also has a reproducibility project: https://r13y.com/

It seems that things must have recently regressed; the page currently says 40% but it used to say 98%.

Java, the most enterprise-y of the enterprise platforms. Practically no Java build system produces deterministic results. Gradle supposedly can, Maven does not, but I've read that the Java compiler itself can't be entirely trusted to produce exact builds soo...

Reply on news.ycombinator.com