Hacker News Reader

Comment by save_ferris | Hacker News Reader

> people were amazed by the productivity gains

Yeah, docker compose is great for running distributed apps locally when everything “just works”. The problem is that it also impedes development when the compose setup isn’t fully optimized.

I work on dockerized rails apps, and whenever a new dependency is added, it can take 15 minutes to rebuild the image, which completely breaks my flow. Docker also creates a ton of data bloat locally, requiring me to run docker system prune pretty regularly. Setting up a dependency cache isn’t completely straightforward either.

I still think docker compose is worth it, but it’s a deceptively complex beast to get right, and most companies I’ve worked for don’t take the time to get it absolutely right. There’s definitely a cost to using it that people tend to overlook.

View on HN

Split the dockerfile into 3 parts. builder image for gems. builder image for assets(nodejs and stuff). copy code from disk, gems from image one, assets from image two, into the third image. you will end up saving HUGE time

Thanks, I’m gonna try this.

Do you have an open source Dockerfile that demonstrates this pattern? I've used builder images before, but not multiple.

The relevant search term is "base image"

Just remember to have a process to rebuild your base image at least once a month so you pick up latest security fixes etc.

Or search for multi-stage builds

First, make sure you only copy your Gemfile and maybe Gemfile.lock before running bundle install. If you aren't that could explain the amount of bloat since every build will create some large layers with no chance of re-use.

You could try making your own base image that installs your gems. Eg, copy your Dockerfile to Dockerfile.base. Remove everything after the bundle install, and build to say app-base:latest.

Then change the FROM in your Dockerfile to app-base, and remove any OS package installs. Keep the bundle install; it will use the layer from app-base if unchanged, or do an incremental install if not.

This will reduce the amount of data bloat too.

You could try using guard-bundler or something similar to further streamline the process when adding gems

I'd suggest using targets within the dockerfile combined with buildkit. Targets allow you to build multiple images from one dockerfile or have multiple distinct stages that depend on one another. Using buildkit means you can build target dependencies in parallel.

Personally, I found having multiple intermediate dockerfiles more confusing than helpful, so having an easy to follow, centralised build made life a lot easier for me, especially if you produce multiple images that are pretty similar at the base.

At my current company, since I'm using gRPC, the build contexts for my containers are annoyingly large and I'm definitely feeling this. It's a massive pain and really annoys me but If I had to choose between 40 minutes of stressful debugging of someone else's config files vs 40 minutes of waiting for a command to finish, I'd take the latter.

Long term I'm planning on moving these builds to bazel and taking advantage of incremental builds for my entire monorepo. I'm in the final stages and just validating some stuff is in place before I do that. In the short term I've mitigated this by adding in heavy unit test coverage. Backend devs can basically run the unit tests for most services and are confident the service is working as expected and also enforces any/all security/auth checks. When developing I don't even start up the containers. I boot them up once at the end just to check my work.

Why not just have a shared volume for your dependencies? That way nothing needs to rebuild.

Then just have a different Docker-compose config for prod

The main reasons are:

    1. My applications have no dynamic "linking"
    2. The containers are distroless
    3. Services can be built independently
    4. I run the same exact sha in dev/staging/prod

To do what you're suggesting would require giving up one of my features which for me is not worth the trade off of a one-time setup 40min build of 20+ services (~2min/service). The last time I paid that 40min cost was ~2 months ago when I reformatted my hard drive.

The better solution for me is looking into a build system that can support my different language and tooling choices which happens to be things like Blaze (Bazel/Pants/Buck/Please).

If I understood correctly, this means that every time you add a new dependency, it takes 15 minutes to rebuild the first time you `docker-compose up` right?

I assume this is due to docker downloading the related image and running the container.

If this is the case, isn't this orders of magnitude faster than adding the dependency manually. And aren't the gains are even more significant when we factor the fact that most of the cases dev and prod environments are identical? Or am I missing something?

> this means that every time you add a new dependency, it takes 15 minutes to rebuild the first time you `docker-compose up` right?

Yes

> If this is the case, isn't this orders of magnitude faster than adding the dependency manually.

No. When developing a rails server running locally, you just need to install the new dependency via bundle install (usually takes just a few seconds) and then your server is ready to run again. With docker, the image needs to be entirely rebuilt, which is a much slower process, even with good caching. The difference is that docker treats the build as ephemeral and willing to be thrown away and rebuilt, as opposed to running directly, which simply augments the existing environment. It’s a much, much slower process to rebuild via docker.

Adding Ruby dependencies (gems) is pretty trivial. And the Ruby package manager (bundler) only installs the new dependency if you do it on your own machine. In Docker it will do all of them.

It's worth noting that bundle uses a Gemfile, so it's not installing each Gem as a seperate step.

You can also skip the docker rebuild and map in the build directory from your host system as bind volume instead.

At work we use docker-compose for legacy Rails (3!) development (we deploy on AWS ECS using Cloudformation templates).

I wasn’t involved in the setup but adding a new gem or upgrading one is very quick. We use a mounted volume to hold the gems as seen here: https://github.com/concord-consortium/rigse/blob/master/rail...

Reply on news.ycombinator.com