Hacker News Reader

Comment by e1g | Hacker News Reader

I recently moved our GutHub CI to a self-hosted runner and it reduced CI times for all jobs by ~5x.

One frustrating part is that a single GitHub runner can handle one job at a time, and our repos have 4-8 highly parallelized jobs. So we need multiple runners. To do this in an economic way, I made a Docker image and run 10 instances of it a single dedicated host from Hetzner. For ~$50/mo, we have unlimited CI and job run as fast as theoretically possible.

The CI box has no inbound ports open to the internet, and the OS is set to auto-update, so the maintenance burden is low.

View on HN

Developer velocity has been my project for the last quarter at work: we just switched from GitHub runners to Hetzner machines, and saw similar improvements to average build times. Between self hosting our runners and switching to bazel which caches aggressively, we've driven down average CI test time from 23 min to 2m40.

That sounds more like expected than an issue.

You might not want to overload one instance.

And running it like you do, is a no brainer anyway

Executing a GitHub runner in a container is a no-brainer, but it's still easy to overload the _host_ with too many jobs/containers, so managing resources is always up to the operator.

I understand why GitHub has this constraint - to avoid clashes between jobs. If multiple jobs (from the same repository) are executing simultaneously within the same stateful environment, they are more likely to clash over shared resources (/tmp, cleanup tasks, database names, etc.). However, even if my jobs are clean and idempotent, GitHub is nudging me to think about runners as "VMs" rather than "containers" (because often CI jobs involve their own containers, and docker-in-docker is a pain), and "self-hosting a bunch of CI VMs" becomes expensive rather quick.

well, GHA supports running actions in containers, which is the best way to control the environment your CI/CD runs in, so putting your runner in a container won't work for everyone.

but, if it does work for someone, doing what you've done will give a much better experience to their developers.

my employer uses single-user VMs for it's runners; it works well, but sometimes actions invocation is high and it can take a few minutes for a runner to come around to taking my job. that would be much less of a problem with dockerized runners.

Can your jobs use dockers or do you run into the docker-in-docker issue?

No Docker for us. I baked all dependencies (DB/Redis/etc) into the main image with the GitHub runner to maximize performance. This works well for us as a relatively small product team that tightly controls all environments. Modularity and flexibility come with a cost, and it's not worth it (for us).

One alternative is to use Firecracker on the CI host and run entire VMs for jobs (still quick). However, I am not comfortable with that stack to do "set & forget" implementation with no ongoing burden.

is your dockerfile anywhere I can see it?

Here it is https://gist.github.com/eugene1g/9e3580a927de9f763703646284c...

thank you kindly.

Reply on news.ycombinator.com