Hacker News Reader

Comment by throwaway9870 | Hacker News Reader

> If you run docker build twice with the same Dockerfile you might get 2 images that behave in different ways. For example, a third-party package could silently be updated and cause breakage. Aggressively pinning dependency versions helps, but doesn't completely prevent this issue.

If "aggressive" means fully, then why doesn't that fix the issue?

View on HN

Author here. In our case, we had a large base Docker image called Polygott (https://github.com/replit/polygott) it pulls in dependencies for 50+ different languages from various repositories. We would pin things where possible, but its still very difficult to ensure reproducible builds.

Additionally, docker builds have free access to the network to do anything it would like. Nix goes to great lengths to sandbox builds and limit network access. Anything accessed from a network requires a pinned sha 256 hash to ensure the remote data hasn't changed. (https://nixos.wiki/wiki/Nix#Sandboxing)

That makes sense. I think the real issue isn't Docker vs Nix, it is that some package managers are almost impossible to use to build reproducible images. I worked with debootstrap 10+ years ago trying to script reproducible builds and found it exceeding hard. Gentoo made it almost trivial (Google used it for ChromeOS so perhaps they felt similar). I will look into Nix.

It appears that with the proper package manager support, Docker would be fine?

I come from a hardware background and seem to be a lot more paranoid than most software folks. I would struggle to trust a build where so much is not pinned.

Why do you need reproducible builds for Docker? The whole point is that you build it once and then you use that container on as many servers as you want.

What happens when you need to update some dependency within that image? Now you have to do an image rebuild. If you're lucky only the top-most layers will be rebuilt and the base layers stay the same, if you're unlucky nearly the whole image is rebuilt. Usually we just want to update some subset of dependencies, but during the rebuild some other dependencies may get updated unintentionally (eg if they aren't pinned to a particular version). For most, this may not be an issue but at Replit, everyone's projects use this base Docker image. Unintended updates can cause breakage for our users.

That's not really what a reproducible build is though. Reproducible builds are you get the exact same thing from your build script today or three weeks for now. Getting unexpected changes with an updated dependency is a different problem than not having a reproducible build.

Fair, but its still a real issue and solved in a similar way: Nix has finer grained reproducibility -- not only at the environment level but also at the derivation level. Being able to pick and choose which dependencies to update while ensuring other packages are left exactly the same is valuable to us.

A lot of package repositories prevent this now, but theoretically your package repository could allow a new version of the package to be uploaded with the same version number. Docker tags have this issue in fact, so even your base image could change underneath you.

I can't imagine using a package system for building images that doesn't allow a full local cache of exactly the packages I want installed.

I mean sure, you can absolutely retain your own docker repository, your own NPM repository, etc and then configure them in your Dockerfile before installing packages. Pretty much every technical problem has a technical solution, but it's more effort that "just" fully pinning your versions.

If there are too many packages to pin, then I assume there are too many package to audit changes in for each image build you do? If that is true, how do you have any confidence that the image is going to function correct?

I think we're talking about two different things. I'm thinking of pinning packages as say, my npm.lock specifies I will use version 2.2.19 of tailwind css. Or my Dockerfile has redis:6.2.6 as it's base image. Both of those are fully pinned, but nothing stops Tailwind or Redis from uploading new versions of the package and reusing the version number (I believe NPM actually disallows this, Docker hub does not). So my build is not technically reproducible.

I can solve this by maintaining my own Docker registry, and my own NPM registry, but that's more work that just fully specifying versions in a configuration file.

As for auditing changes, well most people don't. They update their dependencies and run their tests and if the tests pass, ship it! Most companies don't have a team auditing every single dependency that's pulled in, much less every update after a dependency has been approved. They simply trust the authors of Redis not to screw them.

It's great if you do, I'm certainly not arguing against it, but it's far from the norm.

You can reference docker images by their SHA digest which is immutable, e.g.:

    docker pull redis@sha256:619af14d3a95c30759a1978da1b2ce375504f1af70ff9eea2a8e35febc45d747

Probably because of non-deterministic builds. Especially if the dependency pulls info from the network at build time.

I guess that would not fit my definition of "fully".

Do people really build images this way? It sounds completely insane to pull packages like that randomly from the Internet.

Both my professional experience, and public examples would seem to suggest that's the norm. Example, here's the official postgres Dockerfile: https://github.com/docker-library/postgres/blob/3bb48045b4dc....

Do you work in an environment that maintains custom copies of every dependency in company managed repos? If so, my experience suggests your the outlier, not the people running apt, npm, etc inside their Dockerfiles.

Not custom copies, but a locked down cache of packages. For Gentoo you can do this by locking the portage tree you use and keeping a copy of the distfiles from the first run, for Python it was a requirements.txt file with a cache of the tar files from PyPi, for go it was including 3rd party code in repo. I don't know what the team did for npm.

It was really nice doing a full image rebuild and knowing the only thing that changed it was you explicitly changed.

I'm genuinely curious about this. How are you distributing these caches so that if I do a build on my machine it'll produce the exact same image as on yours? If I'm understanding what you mean by "cache" (I'm thinking the node_modules folder for example for NPM) it'd certainly work, but it feels like a logistical nightmare to me.

We host our own Maven and PyPi. External artifacts get pulled into the cache and devs get all their artifacts form our cache. If an artifact already exists in the cache then we never update it.

Super easily honestly, one of those things that we never even think about until someone upstream does something that would have screwed us anyway, like republishing a version number

We had a VM in a datacenter that hosted everything and was accessed over a VPN. Again, I don't know the specifics of the NPM setup, but for everything else it was HTTP serving static files right off disk. It was a manual process to add new files, but generally they got added in batches so it wasn't too painful. Gentoo has hashes for everything, so at least for OS packages, you could not overwrite anything without breaking the build and noticing.

Reply on news.ycombinator.com