Hacker News Reader

Comment by theamk | Hacker News Reader

except not really.. their tutorial [0] includes no-compile languages like python and JS, with only non-trivial compilation being and Java... and how do they do it?

    RUN gradle build

At least in this case, Earthly has no insight into how repository is organized, and it _will_ recompile the entire repository with every single commit.

So what's the real value of it? this basically seems like a better "docker build" alternative, giving nicer inter-layer caching. Maybe will save some time on dependencies if they take some time to install and devs were too lazy to set up own workers and used ephemeral runners instead.

[0] https://docs.earthly.dev/basics/part-1-a-simple-earthfile

View on HN

That's step one, 'how to write a basic Earthfile that just does the thing', so they don't delve into the details of caching and how it works, but look at the whole block and not just that one line; it operates like a Dockerfile:

    build:
        COPY build.gradle ./
        COPY src src
        RUN gradle build
        RUN gradle install
        SAVE ARTIFACT build/install/java-example/bin /bin
        SAVE ARTIFACT build/install/java-example/lib /lib

The COPY phases copy files into the build context. If the files haven't changed, then the commands don't need to be run. In other words, if `build.gradle` and `src` are unchanged, it won't run `gradle build` or `gradle install`, and it'll just give you the artifacts from the previous build.

They have a golang example in part three[0], which they redesign to use caching effectively:

    build:
        # Download deps before copying code.
        COPY go.mod go.sum .
        RUN go mod download
        # Copy and build code.
        COPY main.go .
        RUN go build -o output/example main.go
        SAVE ARTIFACT output/example AS LOCAL local-output/go-example

They copy in go.mod and go.sum and then run `go mod download`. If the mod and sum files haven't changed, then `go mod download` doesn't need to be run.

Then they copy `main.go` in, and run `go build`. If the `main.go` file hasn't changed, `go build` doesn't need to be run.

[0] https://docs.earthly.dev/basics/part-3-adding-dependencies-w...

Gradle can do the caching of tasks itself in a level of granularity no CI can. If you keep your build cache "live" in the CI, Gradle knows to only recompile what has changed, and to only run tests which changed (or tests something which changed). I am not sure if many other build systems do that, but this is something I would absolutely expect from the build system, not CI. Do you not expect caching when building locally??

The whole idea with CI doing this is wrong IMO. All I want CI to do is call a couple of commands after setting up the environment (which would include the Gradle cache or whatever to speed up things - no need for your revolutionary CI build system).

A CI system like TeamCity will give you that sort of thing for free though, without needing an Earthfile. It just dedupes builds at the same commit, and keeps checkout directories between builds on the agents, so if the build tool has a cache then it's automatically reused. Works great.

Given that it's hard to understand what exactly this is for? Build systems normally try to avoid redundant work for local purposes anyway. Is it some hack for companies that insist on resetting build environments to zero on every single build, and then decide they want caching back? If so, why not just ... let the builds take place in directories containing previous builds, and fix the problems? That's what we do at my company and it works great (using gradle), there are only rarely problems with unclean builds and it's easy to rerun with a clean checkout if that's suspected to be an issue (the bugs are mostly in our own code which does a layer of caching above the build system anyway).

One of the chief benefits of Earthly is builds that are always clean and predictable. This is especially important on a shared build server.

So they claim. And yet ultimately they do it by caching things, same as every other system.

sure, but in a java project, how likely you are to have a commit with no changes to src/? or for a go project, how likely is it that you want to duplicate you entire dependency tree in a clunky, non-parallel dependency description language, vs just doing "COPY * .; RUN go build" and letting golang figure it all out and build in parallel using all the cores?

Don't get me wrong, there is definitely value in caching dependencies, and I can see Earthly approach help if you have, say, a Python app, with 10 minutes of dependency install time, 0 build time, and 10 seconds of tests - you'll see huge improvement. But you'll also see almost the same improvement if you use Dockerfile on a persistent host instead, and as a bonus you won't have to learn new system.

One place Earthly might shine are monorepos with multiple independent projects that even use multiple build systems... but I am not sure how many of those exist _and_ don't have some other caching solution already.

That only splits dependency-build from app-build. Any change to the app source code still causes a full rebuild, unless you figure out how to share the cache directory between builds (and then parallelism gets sketchy).

Generic CI just simply can't parallelize+cache on a level comparable to systems that understand the build itself (Bazel, sccache, etc).

Reply on news.ycombinator.com