Hacker News Reader

Comment by dhruvrrp | Hacker News Reader

Slightly off-topic, it really irks me when people take about reproducible builds/environments while using docker, while having something along the lines of apt-get update/pip install in their dockerfile.

Like that completely destroys the reproducibility of your container!

View on HN

I can't speak for others, but, while we don't pin versions in the Dockerfiles for our build container images, we do version the images themselves.

Reproducibility is a continuum, not a binary. We've chosen a point on that continuum that we believe gives us the best trade-off between reliability and maintenance effort. 100% from-the-ground-up reproducibility would be ideal, of course, but there's a cost-benefit tradeoff, and we're not being paid to be perfectionists.

I mean in a sense, the fucked up behavior of system package managers relative to the actual concerns of building and distributing software to the masses is the reason we need Docker to sandbox environments in the first place. It's 2021, there is next to no reason why the default behavior is installation to /usr/lib with shared objects that are rarely shared, with global access when installed for one application used by one user, and linked dynamically when the object never changes over the lifetime of the running application.

For all its warts this is something that npm gets right. Package management is a tool for software development, not software distribution to end users.

You seem to be missing the point of what we call a "Linux distribution". Shared libraries are fantastic and everything in /usr/lib should be shared

I don't appreciate the condescension, is it not fairly well known that shared libraries aren't typically shared at all? You can distribute software that is reusable and packaged for many machines (the ultimate sense of "shared" in my opinion) but that's difficult when it relies on distribution-specific nuances and packages.

Do you really enjoy having separate packages for the same software for each flavor of package manager that does the same thing? Wouldn't it be nice if we didn't have to use containerization to distribute the same software to machines running a kernel with a stable ABI?

The majority of "shared" libraries are used by exactly one application: https://drewdevault.com/dynlib

It makes absolutely no sense to pollute a global namespace with these.

Out of 958 libs, 230 of mine on my desktop are only used by one application, according to the results after running the stuff from the link. I'm not disagreeing with you about polluting the global namespace, but at least in my case, it's certainly not a majority.

Except that - Linux distributions aren't solely tools for developing software, and their package managers are intended for all their end users.

That's not my point - dependency resolution is a build time requirement, not an install-time one. Requiring the user's machine to resolve dependencies to install your software is fragile, bug prone, and harms the user experience.

Like it or not, users don't care if your software has interchangeable parts. They care if it runs on their system. The only sane way to guarantee a piece of software runs outside your developer machine is to include its dependencies during distribution and packaging (not refer to them - which is what package managers require). The less sane way is to use containers, but those are required when developers don't package their software sanely.

This doesn't preclude users from installing software or replacing interchangeable components should developers support it. What it prevents is disgusting bugs and workarounds because dev A built on distro B while user C wants to use it on distro D but the packages have to be separately for everyone because the distro package managers don't agree with each other on what things are named or how they should be built.

No thanks. Apps like Firefox should use the libraries - windows, menus, etc. - installed on my system. Please don't ship your own libraries for this stuff.

Containers are an even more insane approach, so maybe we are in violent agreement.

Both apt and yum support installing to whatever directory you like as long as you have write permissions there.

Not really. They support dumping the package contents to an arbitrary directory, but you can't actually run the software from there without either the software having been written to support arbitrary paths (almost none is) or using namespacing and chroot to build it a sandbox wherein its baked-in paths actually work.

Contrast that with something like RiscOS AppDirs, classic Mac applications, or Next/Mac Application Bundles.

You very much can run the software from those directories, what are you talking about? Those paths are handled by the OS itself as well through LD_PRELOAD and PATH.

If what you're saying were true, unmodified software wouldn't work in a Docker container, either.

EDIT: Here's the exact command to do what you're saying isn't possible: apt-get download package; dpkg -i --force-not-root --root=$HOME package.deb

I was going to give you the benefit of the doubt and actually try this so I could show you that you are wrong, but I couldn't even get that far because dpkg always complains it is unable to access the dpkg status area. So clearly this is not as trivial as you make it out to be. I suspect because it expects a full filesystem in $HOME with its status file in the appropriate place. In other words, it is expecting a whole separate installation to be under $HOME.

Regardless, lets assume it did work. Here's what it would do: unpack the package replacing '/' with '$HOME' in the destination paths. That's it. That software will not magically be able to find its associated libraries and configurations without the user mucking with environment variables at best, or chrooting or sandboxing such that $HOME appears to it to be a wholly separate installation.

That's not how sane systems do this sort of thing. I have been trying to do this sort of thing in Linux for pretty much as long as I have been using Linux because I loathe the way Linux installs software, and in 20 years it has never been straight forward. AppImage is a close as we get and software needs to be carefully built and packaged for that.

> If what you're saying were true, unmodified software wouldn't work in a Docker container, either.

> [...] or using namespacing and chroot to build it a sandbox wherein its baked-in paths actually work.

Point taken, though there's no reason the package manager couldn't change the environment variables based on the directory it's told to use for the install. The pieces are there, but nobody's really taken the time to put them together. For the filesystem, some simple bind mounts would probably suffice, though you'd still run into some permissions issues I imagine.

EDIT: It seems to be significantly easier on dnf, to the point that it could be trivial to add full support for home-directory installs.

"support" and "default" are very different things. Especially with package managers like homebrew, which comes with a loud warning that non-default (root) paths may not be supported by dependencies.

It also doesn't really, since this is a system level issue. Applications need to package their dependencies, not the other way around. Dependencies form graphs, not flat lists. The existence of a global cache of libraries shared by all programs is a total inversion of requirements.

Depends on how you do it, right?

For instance, we create base images configured with SDKs, libraries, frameworks, configurations, binaries, etc...

Those base images are then built, versioned, tagged and then pushed to our container repos ready to be used by developers, CI/CD, etc...

Images based on these base images never need an apt-get, pip install, etc... If there is a dependency missing, updated needed, etc... we'll create a new base image with it, following the steps above.

I would love some constructive feedback.

This is what I think is the most practical approach. Pinning down ALL your dependencies to the exact version is much harder than it sounds. What you’re describing sound like the way Jib [1] does it. The pictures in this [2] blog post help visualize it.

The reason I like the approach you describe is because it keeps things simpler at the start of a project and consistent across most projects.

I also think it makes sense to have those support containers build on a schedule. For example, you build your build/CI container weekly and that’s the CI container for the week. On demand project builds use that CI container which has all dependencies, etc. baked in.

It would be nice if CI systems would let me explicitly tag builds as (non)reproducible.

1. https://github.com/GoogleContainerTools/jib

2. https://phauer.com/2019/no-fat-jar-in-docker-image/

That solves the problem one way, but I don't know if I would call it reproducible. You're working around the fact that it's not reproducible by only doing it once. You don't get the benefits, e.g. everyone who uses your images has to trust that you built them the right way.

I guess what I mean is that everything built using this base image should be reproducible. There is no reason (hopefully) to reproduce the base image. Any changes to the base requirements (apt-get, pip, etc) requires a whole new build and results in an entirely different artifact.

And just to be clear, I'm not building (no human) the base image. The base image is also created within it's own build pipeline that has all of the necessary things to track its materialization and lineage. Logs, manifests, etc...

Once the image has been thoroughly tested and verified (both by humans and verification scripts) each time a change is merged, the git repo is tagged, docker image is built and tagged and then pushed to the container repo.

Perhaps you could explain what you mean by the other way? Why would you ever need to recreate the base image? Perhaps if the container repo dropped off the face of the earth and had to be created from scratch?

The industry seems to have adopted “hermetic” as a word that describes truly reproducible builds, while “reproducible” has a lower standard. In many cases, it seems to be used to mean “not dependent on local build environment”.

Yea - I get where you're coming from - you can't rebuild the container, but each team member can reuse the same container - which is at least a step towards being fully reproducible.

We don't rebuild image every time. We store images in GitLab registry. Also, just because we use apt-get to install python3 doesn't mean build isn't reproducible. The actual toolchain and sysroot are highly version controlled and python is just used to kick off the build script so it almost doesn't really matter which version python is as long as it's backwards compatible with the build script.

We tried this in our research group, and found that issues come in when it gets to things like...

Oh no! CMake is too old a version to support a dependency we have to build in the image construction. So we better pull in a version of CMake from a PPA which is community maintained, and build it from source/etc.

If I use tools that can give me reproducible builds, like Bazel or Nix, I won't need or want to use containers for development.

I'm interested to know your alternate solution. How would you recommend getting the required libraries into your image?

I guess the OP meant such that apt/yum/dnf/whatever runs every time the image runs, rather than just once when it's built. Not that that's something I see very often, mind.

Isn’t the standard approach here then to derive from base images, which have exact versions? Being honest, I’m not a Docker/VM legend but I’ve seen a few attempts at managing this, and base images was one of them.

Personally, I don’t see the issue with it if you’re at least being a little careful— don’t make obvious mistakes like installing latest/nightly packages automatically, etc.

Yes, that's the standard approach, but the base images are frequently updated. If you really want to pin at a specific image, you need to specify the image hash, rather than using the "latest", or even a version tag (e.g. "2.1").

In your image that extends from the base image, you'll typically update the package repo cache (it is typically cleared after building the base image, to reduce the size), then install whatever packages you want.

Like you, I don't see a particular issue with updating system-level packages - especially from a security standpoint.

I never understood why commands like apt-get don't take a version with the dependency name.

They do. You can do `apt-get install package=version`

As far as I'm aware it's supported (something like `apt-get install virtualbox=5.0.18-dfsg-2build1 -V`). It's just not commonly used because you usually just choose a distro with the desired update granularity (whether you want to the newest version out there or a consistent version with backported security fixes, or something in between).

Keeping apt-get in our container builds is vital, since it helps keep CVEs out of our containers. We can do automated re-builds of all of our containers weekly and (typically) it lets us keep on top of the CVE game.

And by that you mean that the version of those deps aren't pinned?

You depend on what the repository serves for "apt-get". You may pin it, but it still doesn't guarantee you're going to get the same version if it was replaced on the repository without changing the version.

At least for python, just pinning the top level dependencies are not enough. if you pin tensorflow==2.4.0, it doesn’t pin its required packages, rather it just defines a range. An example would be tf will try to get wheel>=0.26 (which was released in 2015 and is currently on release 0.36.2)

Whats the correct way to get libraries into your docker container? We have plenty of apt-get / pip calls in the dockerfiles at my work (set up by someone else).

Not if you specify the versions to install, at least

Reply on news.ycombinator.com