Hacker News Reader

Comment by dangoldin | Hacker News Reader

Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.

View on HN

Systems like these should have an escape hatch of some sort. The key part is that it needs to be auditable.

Anything you do in CI should be possible outside of CI, at least by some subset of users.

Yea - definitely. Just not ideal and something that needs to be built out, tested, etc.

Yes, it is easier said than done. At my company we use buildkite and many people wrote scripts that simply fail outside of buildkite.

GitHub actions is even worse, it seems like it was designed from the ground up to create lock in.

Nix helps a bit on the bootstrapping and dependency management problem, but won't save you from writing a script that is too tightly coupled to its runtime environment.

I've seen numerous "escape hatches" over the years that actually just turned out to be painted on the wall. If you actually tried to use them. No one ever does though.

I don't think its malice. I just think its pretty uncommon for anyone to intentionally back out of a structural tech decision so it gets forgotten about and remains un-battle tested.. That or the timeline is longer than SaaS has been around.

This is why I personally like to use none of the CI features, and mostly use it like a shell script executor. Images? Stick to OS images only so that you can easily spin them up with `docker run` locally. Artifacts? Read and write them into S3 buckets and avoid the native artifact features.

This is obviously more difficult in the Github actions ecosystem, but I have mostly used Gitlab CI so far. My CI pipelines mostly look like this:

    image: ubuntu:24.04
    before_script:
        - apt-get install ...
    script:
        - ./ci/build-project.sh
    after_script:
        - ./ci/upload-build-artifacts.sh

I've run into a scenario where one of our rarely used environments needed a hotfix and the GitHub action we used to deploy there was broken. Was easy enough to translate GitHub action steps to shell scripting for a quick, manual deployment.

Reply on news.ycombinator.com