574 karma · joined September 6, 2017
Blog at https://blog.kylegalbraith.com
Previously at: Era Software, Thorn, Zapproved
Previously built: https://ultimatefantasysupercross.com/ https://kylegalbraith.com/learn-aws/ https://kylegalbraith.com/learn-by-doing/
kyle@depot.dev
The registry cache idea is a neat idea, but in practice suffers the same problem.
We came to the same conclusion and built Depot around this exact workflow for the Docker image build problem. We're now bringing that same tech into GitHub Actions workflows.
You've hit on all the main points regarding Docker image cache in GHA. Persisting the massive layer cache over networks is incredibly slow and has weird limits (like 10GB per repo). We persist the layer cache to ceph volumes and orchestrate your cache so it's immediately available across builds with our first service, accelerated container image builds. Our GHA runners run right next to that same infra, so you don't have ingress/egress. All that can be hosted in your own AWS account (we're also open to running that in any general compute environment for folks who need it).
I also love seeing stupid simple HTML sites like this. It's hideous to look at but it serves exactly the purpose it's there to do. It's also really fast.
We're working on another project in this realm that might interest you. Happy to send over more details via email if your interested in better build APIs. My contact info is in my profile.
It's not exactly what you're asking for as it relates to running containers via runc. But this walks through how OCI layers are actually built up behind the scenes. In case it's helpful: https://depot.dev/blog/building-container-layers-from-scratc...
We've optimized BuildKit for remote container builds with Depot. So we've added things like a different `--load` for pulling the image back that is optimized to only pull the layers back that have actually changed between the build and what is on the client. We've also done things like automatically supporting eStargz, adding the ability to `--push` and `--load` at the same time, and the ability to push to multiple registries in parallel.
We've removed saving/loading layer cache over the network. Instead, the BuildKit builder is ephemeral, and we orchestrate the cache across builds by persisting the layer cache to Ceph and reattaching it on the next build.
The largest speedup with Depot is that we build on native CPUs so that you can avoid emulation. We run native Intel and Arm builders with 16 CPUs and 32GB of memory inside of AWS. We also have the ability to run these builders in your own cloud account with a self-hosted data plane.
So the bulk of the speed comes from persisting layer cache across builds with Ceph and native CPUs. The optimized portions of BuildKit really help post-build currently. That said, we are working on some things in the middle of the build related to the DAG structure of BuildKit that will also optimize up in front of the build.
It’s really helped me gain a deeper understanding of the language and feel more confident in conversations.
The best part is that it pretty much just works. Very little babysitting with the exception of the occasional fs trim or something.
It’s been a massive improvement for our caching system.
Cool idea. I'll be watching to see how it evolves.
Dive also inspired us to make it easier to surface what is actually in your build context, on every build. So we shipped that as a feature in Depot a few weeks back.
LLB supports 6 core operations:
- SourceOp loads source files from a Docker image, git repo, or local build context (e.g. FROM) - ExecOp executes a given command (e.g. RUN) - FileOp creates, copies or removes files and directories (e.g. COPY, ADD, WORKDIR) - MergeOp merges its inputs into a single flat layer (e.g. COPY --link) - DiffOp produces an OCI diff, e.g. "layer", for its inputs (unused in the Dockerfile frontend) - BuildOp evaluates its input as additional LLB operations to add to the graph to allow for dynamic build graphs (also unused in the Dockerfile frontend)
With the Dockerfile Explorer, we run the Dockerfile frontend[1] that BuildKit uses inside of WASM to parse and produce the LLB output locally in your browser. We then embed the Monaco Editor so that you can change your Dockerfile to see how it impacts the LLB output that BuildKit will use to build your Docker image.
You can see a quick video and read more details on how it all works here: https://depot.dev/blog/dockerfile-explorer.
We'd love any feedback or ideas folks would like around this type of tool!
[0] https://github.com/moby/buildkit#exploring-llb [1] https://github.com/moby/buildkit/tree/master/frontend/docker...
I've never been diagnosed with ADHD but when I read pieces like this I always walk away feeling like I check most of the boxes. Has anyone ever encountered the same and been prompted to get help? Asking on HN probably isn't the best idea, but sometimes I really struggle with certain flows and the way others work that seems to run counter to my peak performance.