Hacker News Reader

Comment by jakobegger | Hacker News Reader

I recently posted an „Ask HN“ asking for a flexible CI system. Most of the suggestions were proprietary hosted SaaS solutions.

I was sceptical, and things like that keep me sceptical. Sure, setting up a CI system yourself is maybe a bit harder than using a hosted solution. But if the provider decides to change their offering, you are in big trouble. You‘ll need to spend a lot of time switching to a new sevice, migrating all your data and processes. And once that is done, you can start to retrain all your employees.

And you always have to be ready for that scenario, at two months notice....

View on HN

I've built several Jenkins-based CI systems at work. Once you know how to not shoot yourself in the foot, it's really easy. I have 2 cardinal rules:

1) Don't run any jobs on the master box – master is just a web server + master Jenkins app server 2) All build scripts should be part of the project, not configured CI

#2 is the more important. Jenkins jobs at my work used to have tons of configuration and plugins, because the person setting up the CI would see (for example) an "Xcode plugin," and (naturally) decide to install and use that. That was a nightmare because of transient dependencies (and conflicts), and now configurations like "project file name" were spread across multiple jobs. Now we just have similar build scripts in each project, which has the additional benefit of letting developers run builds and tests locally the same as they run in CI. Jobs are tiny, and all the CI configuration has to do with CI things, like schedule, branch, archiving, notifications, etc.

At that point, is there much point in using Jenkins? Aren't there much lighter weight alternatives?

If you know a good one, definitely let me know. Generally the only constraints are open source with a permissive license & on-premises installs.

One benefit is it can already integrate with whatever else is out there. When we switched from GitLab to GitHub Enterprise at work, our CI environment just needed to point somewhere else.

Why did you switch from GitLab? What should we improve?

Nothing! GitLab was great. But we had several GitHub, GitLab, & Stash instances, and there was a push to consolidate onto one platform. Not sure why we consolidated onto GitHub over others, but that was above my pay grade :)

Not a reasonable replacement as maintaining your own mac servers is non-trivial and all the major cloud providers don't have mac as hardware options.

Except the parent comment wasn't asking about a cloud-based replacement. They specifically mentioned setting up a CI environment yourself.

The problem I've seen is it makes sense to the person who set it up but when they leave it's not as easy or obvious for the person who comes in later to maintain it. If you have left behind detailed docs that may mitigate the issue. Also in some cases there is good reason to do it yourself but in other cases some people have a strong NIH syndrome and the value in the end to the company is not necessarily higher with a custom setup. In the end if it ends up costing more to maintain the system it's a net loss.

> Setting up a CI system yourself is maybe a bit harder than using a hosted solution. But if the provider decides to change their offering, you are in big trouble.

I love how GitLab does it. It owns the parts watching your branches for commits and passing instructions to your machine, but the instructions themselves are a clean YAML file in the root of your repo, so if at any point you want to pack things and switch over to another provider it’s all right there in plain text.

Manual setup looked a bit daunting initially but turned out to be not that hard to wrap your head around and totally worth it, simple configuration with one runner has worked like a charm for months. Between ease of setup and zero lock-in this seems to strike a pretty good balance.

Yay, glad to hear that. And we're trying to make initial setup easier with Auto DevOps https://docs.gitlab.com/ee/topics/autodevops/ providing a default implicit gitlab-ci.yml file

Initial setup? I was amazed when I simply put my .yml file into the root of my repo and pushed to gitlab. It just worked and the job was running. What other initial setup are you referring to?

There is Buildkite, which a hybrid server side / client side solution. This has worked extraordinarily because one of the most frustrating parts of setting up a CI service is setting up a long-lived, reliable server.

Buildkite also offers free accounts to open source projects along with awesome email support. After years of Jenkins, followed by a couple of months trying to cherry pick CI parts of GitLab and make them play nicely with GitHub, Buildkite has been a breath of fresh air. Their service has allowed us to version control pipeline configs, test GitHub pull requests, and provide all core features without having to curate a list of random plugins.

Note: I'm not affiliated with Buildkite.

We’re working on making GitLab CI work better with GitHub. I'm sorry to hear it will be too late for you.

BTW for anyone interested: this will be done with a CI only workflow https://gitlab.com/gitlab-org/gitlab-ee/issues/3839

Not a reasonable replacement as maintaining your own mac servers is non-trivial and all the major cloud providers don't have mac as hardware options.

I have used (and set up) Jenkins, Drone and GitLab CI, and the latter two are definitely my preference these days. Unlike Jenkins, the configuration for Drone or GitLab sits in the repo, so it's as portable as the code itself - you manage the CI process through PRs and merges, same as anything. I use GitLab for personal projects, and it's pretty nice in that it's also git hosting, but the runners can be finicky to set up, and even though I've set up a few, it can still be confusing (specifically, tagging runners for specific tasks). In general, I think the most flexible thing is the one that's configured on the fly in the project itself.

This is where open source projects with open governance offer peace of mind over proprietary offerings. Community stakeholders get a say in how the project evolves -- decisions are not made solely on what offers the best value for the platform's investors.

Far more nuance is needed here.

There are plenty of open source projects where the decisions and roadmap are decided in private and based on financial motivations. In fact most of the company sponsored projects are like this eg. Spark, Cassandra.

"open source projects with open governance"

> Spark, Cassandra

I'm a former Apache board member, and I've worked with contributors to those specific projects on governance issues. I stand by what I said.

When it is possible for users to become contributors and gain a governance stake, projects tend to become more dedicated to preserving backwards compatibility. People who are counting on the software get in a position where they can raise a big, loud stink when their interests are threatened.

Sometimes projects break compat anyway, but only after traumatic debate. And depending on how the open governance is set up, it can be nearly impossible for the project to be sold out from underneath its users.

Hard part in this case is not the actual build code or even the CI runners, its actually maintaining a mac server farm for your builds. None of the major cloud providers have mac hardware options.

Buddybuild and very few other CI systems have iOS and Android support.

I’ve been using drone, which is open source.

It’s the only CI tool I’ve found writing pipelines in to not be painful. Need a test DB? The syntax is just docker compose so you already know how to get yourself one.

People who cared about BuddyBuild probably don't care much about Docker or "generic" CI services. There are plenty of those.

The thing about BuddyBuild was that it was "iOS Specific". You could point it at a repo and you would have your project building, signed and ready to install, in minutes. This was pretty unheard of. And still is. Trying to get iOS projects to build and sign on generic CI services like Travis, Circle or Drone means hours of pain to get it going and to keep it going. Been there done that.

What would it take to enable iOS builds in an open source project like drone? It is true that drone is primarily focused on container runtimes (linux amd64/armhf/aarch64 and windows) but perhaps we could have a separate yaml and agent that executes a pipeline on the host machine, perhaps using sandbox-exec for isolation.

Reply on news.ycombinator.com