Comment by mschuster91 | Hacker News Reader

Comment by mschuster91 | Hacker News Reader

Parent Full thread

The problem is you can't (easily...) create a layer atop of a Docker image without pulling all of it before.

Basically, a conventional Docker CI build process looks like the following in what I use (where <tag> is something like branch-foobar):

1. docker login <repo>

2. docker pull <repo>/<image>:<tag> || true

3. docker pull <repo>/<image>:branch-master || true

3. docker build --pull --cache-from <repo>/<image>:<tag> --cache-from <repo>/<image>:<master> -f Dockerfile.<branch> -t <repo>/<image>:<tag> .

4. docker push <repo>/<image>:<tag>

Using --cache-from cuts down dramatically on the build times since (assuming your branch-master tag gets rebuilt nightly) at least the Dockerfile layer steps for downloading the base OS image and installing software can be automatically skipped.

But still, even swapping out the last step in the Docker build makes it necessary to pull the entire image first - assuming your average Java application, that's like 500+ MB for OS+Java JRE+OS dependencies. If you're sure that the change is only in the last COPY/ADD step, you still have to pull the image despite it not being needed technically.

So that's a problem with the builder being ephemeral then.

I am fairly sure a docker layer is just a tarfile. If you're just adding, say, python code and performing no execution I feel like you could create the tarfile and add it to the list of layers.

edit - This seemed dismissive of me. It isn't intended like that, I'm genuinely curious as to whether this is the right approach. The post feels intuitively like it's reinventing some things that exist, but perhaps for good reason. I think it'd be nice if everything else was "just docker". Perhaps I'm missing a key point - it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?

> If you're just adding, say, python code and performing no execution I feel like you could create the tarfile and add it to the list of layers.

Indeed. The problem is, creating the tarfile is "easy" but manipulating the image metadata and getting that pushed to the repository is not - you need to implement that all on your own as Docker tooling is useless without the full image.

> Perhaps I'm missing a key point - it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?

Hard to impossible to do in ephemeral or shared build services. IIRC, you can't add EBS or EFS volumes to Fargate tasks, and with EBS volumes you don't have concurrent access between multiple build runners.

I'm curious to try this now, manifests are just json and I'm pretty sure the registry lets you push these things directly - it's possible with bash alone.

> Hard to impossible to do in ephemeral or shared build services. IIRC, you can't add EBS or EFS volumes to Fargate tasks, and with EBS volumes you don't have concurrent access between multiple build runners.

You can attach EFS to fargate tasks. My more general point there was that it feels like a lot of complexity disappears if you don't have an ephemeral build service and can just use docker as docker.

Something I'd missed in my first readthrough was about using existing running containers - that makes me a little uneasy but "start from scratch" and "update a running system" are fair points to find tradeoffs between them. It all feels like trying to push the tools to work in a way they weren't really intended to.

(author here)

There are two sides to the problem: the build and the deploy.

> it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?

Indeed, ephemeral builders are an issue and this would be an alternative solution. However doing this for an arbitrary numbers of users who want on-demand deploys for a variety of projects is non trivial. Most CI environments give you a "clean environment" at startup.

Even if the build takes 0 seconds, the other part is re-running this docker image (where 100s of MB is unchanged and say the final 1MB layer changed). You need to ship all these layers to a provisioned container and boot it up. Even this is possible to optimize by heavy caching and optimizing the container service - but that's an alternative solution with different trade-offs. With pex, we can update any docker image in-place so it is container service agnostic. It also works with our current service (Fargate) which has famously slow startup times.

>(Fargate) which has famously slow startup times

Haven't tested it in a couple years but start time used to correlate with image size (which makes sense). I think the Fargate VMs don't have very many IOPs so you're usually waiting on extracting tarballs for your container to start

With that in mind, it can sometimes be more efficient to ignore Docker later best practices and just try to smash everything down into the smallest single layer possible

> Most CI environments give you a "clean environment" at startup.

Yeah, it was partly that which made me wonder if just removing that would help a lot. Seems weird to keep a clean fresh CI setup and not have that in production

I understand handing off the scaling issues to someone else. Perhaps this is the difference between the "best outcome" and "best solution" as the latter takes into account what already exists.

K8s does some optimisation on image layers but my knowledge there is fuzzy.

Perhaps this is the best solution given the constraints, but it's a shame that the fully supported layering system in docker needs to be recreated essentially due to two services throwing away lots of useful data. A container runner that stops your container, pulls the layers and restarts it would only have the container startup time itself - which is extremely small - and keep everything separated.

I agree with your sentiment.

K8s may have some more controls for the "incremental deployment" case but I'm less confident about the isolation between pods to run user provided code.

Reply on news.ycombinator.com