We built the fastest CI and it failed

Hi Spivak, You're not Lee Spivak (formerly of Upromise in the early 2000's), are you?

Earthly is a single binary CLI on top of Docker buildkit so it's a bit trickier for them.

The boards are terrible, the UI is hard, the permissions are incomprehensible, it doesn't even integrate that nicely with Azure.

On paper it does everything, it practice it can be made to do everything, but it's like getting citrus drenched sandpaper papercuts every day.

(One of the most recent signs of this crazy zombie state that a recent feature for AzDO was given the absurd only Microsoft could do brand name "GitHub Advanced Secure for Azure DevOps", and it is indeed a bundle of GitHub features several years old at this point finally provided to AzDO users. Other signs include all sorts of AzDO libraries and roadmaps and related repos openly hosted on GitHub with last updated dates in 2019 and 2020.)

From an outsider perspective, Microsoft probably has a lot of sentimental and internal engineering reasons it doesn't want to truly wrap up so much of its operations in North Carolina (AzDO's ancestral home going back to early days of TFS), but the writing seems to be on the wall that what's left is a skeleton crew mostly working for GitHub full time and then sort of applying things part time back to AzDO. It doesn't seem to me to be sustainable long term, and every day it feels more like an Old Yeller situation where Microsoft is just prolonging the inevitable and making the pain worse for everyone involved.

No one EVER should run deploy.sh like how you propose it.

ArgoCD abstracts away access to a k8s cluster (login through dex).

It has strategies of how to deploy k8s resources like through retry, replace (which you would need to write manually in your deploy.sh).

It gives you debug output for your deploy process (yes thats a thing, you do not assume every deploy just works).

ArgoCD also can and should be used to reference secerts from secret store like vault.

And it gives you a history/audit.

But yeah, once your clients are big corporations you'd at least need to know how to implement some kind of cloud mess of authentication provider(s), still, doesn't need you need to infect your own stack with it.

But I think it comes with its own trade-offs. For instance, I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh.

And I lament that people seem to be less critical of entirely opaque proprietary software than of source available software. I think if the only viable choices are nonprofit+open and commercial+closed, then we are going to end up with more closed software that is of higher quality than its open alternatives.

In general my interest in this topic flows from dissatisfaction with funding models for open source software. I really hate that so much important software relies on under- or even entirely un-paid labor to create and maintain it. I'm not under illusions that alternative models are a panacea, but I'm also not satisfied with the status quo. I get paid well for my work, and I hate that so many people whose labor I rely on aren't getting paid well for theirs!

But is it, though?

I mean, are any of the competing CICD services noticeably slow, specially if we focus on the system's overhead?

I don't think there are, at all.

We're talking about at best millisecond differences in workflows that take minutes just to do the things we tell them to, such as setting up a build environment.

Flamebox almost got this right, but groups by wall-clock instead of git sha:

https://flamebox.sourceforge.net

The screenshot is most of what there is to see. Clicking a test run opens the test log.

Does anyone know of a currently-maintained alternative with a less-bad dashboard?

> Earthly gives you write once, run anywhere build consistency, making it super easy to reproduce CI failures on your local computer.

With Earthly you just... write your Earthfile. You get everything for free. Caching and parallelization _also_ works locally, so I see that speedup in development.

It can be papered over, but intelligibility by more junior engineers is one of the reasons I'd opt for GHA over something like Jenkins.

Or, you make reasonable assumptions that are correct in most cases, and rely on the user to provide hints as to when the cache should be invalidated.

Assuming your code, environment variables, and `args` are all factored into invalidating the cache, there are few situations in typical builds where using the cache is a bad idea. The biggest ones I can think of are if your build system is going out and fetching data itself to decide what to do, e.g. if you're downloading source code (e.g. a shared library) from an external source which may have updated it.

That said, if you're just arbitrarily pulling down whatever the latest build of libcurl is or something, you're going to have a bad time eventually anyway, so you should either put a static version into an arg or just put the download into a separate step; at which point the cache will correctly invalidate when it's changed.

Maybe if a build is only valid for 90 days from the build date or something (e.g. a preview build) and that's built into the binary at compile time, but in that case you can pass that in as an arg anyway and then your cache is going to be invalid every morning regardless.

I'm not going to say it's never an issue, but the same code with the same environment and the same arguments, built on the same platform with the same versions of the same dependencies, using the same versions of the same tooling to run the same commands and copy the same artifacts to the same locations should very nearly never produce a valid output now which is not an equally valid output in the future unless it is fetching and acting on external information (which can be easily parameterized).

This made debugging so much easier for these issues. They're bound to happen, because we're all human and bugs are a thing. Automatically being able to audit the build as step 1 makes this easy to spot

I have to be honest - MS Build does not scale well. Companies use it because it’s the default.

    build:
        COPY build.gradle ./
        COPY src src
        RUN gradle build
        RUN gradle install
        SAVE ARTIFACT build/install/java-example/bin /bin
        SAVE ARTIFACT build/install/java-example/lib /lib

They have a golang example in part three[0], which they redesign to use caching effectively:

    build:
        # Download deps before copying code.
        COPY go.mod go.sum .
        RUN go mod download
        # Copy and build code.
        COPY main.go .
        RUN go build -o output/example main.go
        SAVE ARTIFACT output/example AS LOCAL local-output/go-example

Then they copy `main.go` in, and run `go build`. If the `main.go` file hasn't changed, `go build` doesn't need to be run.

[0] https://docs.earthly.dev/basics/part-3-adding-dependencies-w...

I once worked on a system where we had excellent test coverage and all our unit tests ran in under a minute. We worked hard to keep it that way, because it was a huge boon to productivity, especially when improving design issues. It also enabled frequent commits and merges, which made collaboration easier. It was a great experience.

But yes, from the business perspective, the leading edge is a pretty small audience. (Circa 15%, per Moore.) And the bottom half of the audience by competence, which Moore called the Late Majority and the Laggards, can be well more than half your revenue in a SaaS context because they will create problems for themselves (like slow/buggy builds) that mean higher per-user revenue. And they may also have special feature needs where they'll pay more rather than clean up their messes.

In that context, doing the right thing for your business can be very different than doing the right thing for users. Sort of like a physical therapist who makes most of their money from renting crutches and in-home meal delivery, and so doesn't push people to actually get better.

I have been using CircleCI since the beginning and the only breaking change I saw was when they introduced workflows. This didn't need any changes to the difficult parts of my own code and was mostly copy-paste.

I believe that my configs are fairly complex using almost all existing features across many different projects and build targets.

or if they do, they're hosting their own.

if you're selling a hosted CI platform, your sales pitch is "pay us $$ instead of paying your staff $$$ to do devops". if your pitch is "pay us $$ and then also spend $$$ on staff hours to manage our service", it's very unlikely the CI is doing anything useful enough that the staff couldn't just run their own thing in-house.

it doesn't take much configuration to be more difficult than an EC2 instance running a script on a webhook from your version control.

You should at least read and understand the paper "Recursive Make Considered Harmful" before attempting to replace make with something "better":

https://aegis.sourceforge.net/auug97.pdf

Most people use make incorrectly (and it sounds like the system you describe makes most of the classic mistakes). The paper I linked explains how to use Make's templating language to help it scale to complicated setups.

Here are a few critiques of taskfile (from skimming their documentation):

- The syntax is incredibly verbose and weird. For instance, you have to write a tree of yaml nodes to loop over a set of files. Make's syntax is weird, but at least it is not verbose.

- The documentation makes no mention that I could find of deriving a set of tasks from a list of files (Make's $(wildcard) macro, or stemming rules like "%.o : %.c"). So, I guess the taskfile will end up being huge (like the size of autogenerated makefiles from autoconf or cmake), or you'll end up using additional ad-hoc build systems that are invoked by your taskfile.

- From the documentation, they haven't thought through having more than one project per unix account "When running your global Taskfile with -g, tasks will run on $HOME by default, and not on your working directory!"

- They seem to have partially reimplemented the bad ideas from recursive make: https://taskfile.dev/usage/#including-other-taskfiles

- I suggest taking an idiomatic but non-trivial bash script (which is 99% of what make is invoking, and that taskfiles support), and then trying to port it to python directly. There's usually a 10-100x line of code blowup from doing that, and the python thing usually bitrots every year or so (vs. every few decades for shell).

Makefile syntax may have its problems, but nothing is as bad as YAML syntax.

YAML is worse than Makefile syntax on every criteria you may consider.

However if a different team has made a good ci system they don't need to know how it works, just enough to diagnose a build error they didn't see locally. Likewise, someone needs to know the details of git, but there is no need for everyone to know the details of that mess (git is a particularly bad version control, I wish the world had settled on something else - there are several better choices) so long as you can do the basic commits, merges and checkouts. Most senior developers need to know how to add a text file to cmake (or whatever your build system is), but there is a lot more you can do with a build system that only a few need to know.

Senior devs need to be great at the problem domain they are in. That is normally the subsystem they write code in, and good architecture therein. They need to be able to figure out all the tools you list and more, but becoming experts in them all is a waste of time if enough others already are.

As a senior developer myself, I approve this message!

It worked relatively well for somethings, namely builds and focused watch mode (like running watch only on apps via the build tool, e.g. vite or webpack). I don't like that they punted their remote cache to basically be "Sign up for this vercel service or read our API contract, which BTW can change without notice".

It also lacks a smart workspace watch mode, which is frustrating. This is something that wireit in particular does really well. If you look at the GitHub discussions / issues for turbo, they stated that they're effectively only going to implement a watch mode when `turbopack` is ready, because they want to integrate it into turbo in some way.

I don't think I had issues with the cache otherwise. You can, for instance, just keep an artifact in your CI of the turbo cache (its a `.turbo` directory in the root of your repo) and while not shared with your local, it works fine enough.

It didn't have any smart masking of logged output either, so you have to be consciously aware of what gets logged to the console. Not usually an issue, but works well enough.

To be honest though, developing against apps was okay, but if you needed distributed watch mode across apps and libraries (say, a design system or something) it was awful, because turbo doesn't have its own smart dependency graph based watch mode startup, you just end up starting watch mode across all your dependencies with mixed results. For instances, it would trigger builds that could easily go stale or have issues if not built in the correct order, which you completely forgo because you have to use `--parallel`[0] to use watch mode. I also had dependency chain issues with HMR due to out of band builds happening etc.

You can hack around this with things like nodemon, but its really a terrible experience.

It also lacked proper support for phased commands, you had to hack it via pipelines which meant you didn't get 100% unique distributed tasks, which could become an issue with caching if you aren't aware of this. (using && in a script entry is not a substitute here, and declaring dependent pipelines felt like you were doing it wrong, not to mention, added overhead in some scenarios)

All in all, it was fine, but I like the wireit model a bit better.

Turbo also does nothing around module boundary enforcements. This is one thing NX and Rush do well if you use tags. wireit doesn't have this concept on a allow / deny level, but it does have it on a explicit dependency declaration level. Turbo has neither.

I'm a bit wary of Vercel. They way they treat their open source projects and community feedback sometimes give me pause. I'm sure they're great people doing their best, I have concerns the same as I do with Google here.

[0]: https://turbo.build/repo/docs/reference/command-line-referen...

Docker maybe a little more love-hate but I think that has a lot more to do with the politics and salesy side of things and "Who owns what"