Show HN: New approach for CI/CD/compute system | Hacker News Reader

Show HN: New approach for CI/CD/compute system

egdaemon.com

22 pointspandiculation5 comments

I've been tired with the current options on the market for awhile and decided to do something about it after the running into the disaster that is MLOps at my last two startups and having to manage a multiple operation platforms both for my fellow ML engineers, the general application CI/CD and orchestration layers while simultaneously building the application itself.

Its still extremely early for the product but its functioning and is well on its way. I'd love feedback on the approach and peoples thoughts on the problem space. Personally my irritations have been in the poor tooling, slow iteration cycles, an the amount of manual setup that's needed just for a basic system with reasonable performance.

I also wanted something that had a price point that is reasonable for solo projects, as I believe that's where most of the magic happens in our industry and I wanted something that supported those of us in that space while making it easier to contribute to projects by making it basically a single command to spin up the development environment.

I've been pretty happy with the outcome so far, there are still a bunch of things we need to fix in upstream systems to really make it shine. But those issues are fairly straight forward technically just will take some time to get through the human portion of those issues.

Open articleView on HN

As someone who lead this space at multiple FAANGs, just use Bazel or Buck2 and have whatever CI call it. CI should be one line(-ish) calling a local tool. "Fat" CI is an anti-pattern. The problem is not the CI tools you mention are bad and the only thing available, the problem is that newer tools and methods solving these issues are not known or adopted.

And if those don't work for you, build custom on something like Temporal.

Totally agreed on this general pattern, even you're not using Bazel or Buck2.

In my experience working in large organizations - if a developer can't easily reproduce what the CI/CD pipeline is doing, then many won't try. That leads to a lot of "push-and-pray" magical thinking, minimal local testing, and a lack of understanding about what the pipeline does and doesn't do. Across a large organization, this can also get expensive if you're using cloud-hosted build runners.

Instead of putting a bunch of build logic in your pipeline, try to put it in a local script instead. Or better yet, wrap your actual build steps with a top-level Makefile (which can then call bazel, or cmake, or whatever). Design your pipeline to run this target, and keep the pipeline logic to an absolute minimum.

Your pipelines will become small, and easy to reason about. Your CI/CD setup will be more portable between different products (Jenkins, Azure Pipelines, Buildkite, etc). And your developers will be able to use the same workflow as the pipeline, leading to fewer "but it works on my machine!"-type problems.

bazels for building. and by all means use bazel if thats your jam, not what EG is about directly.

Personally, I never saw bazel as terribly useful with the languages I use since they tend to have decent build/dependency management built in. (aka: it was a solution made for c/c++ and python which have terrible build systems)

Nor did I find it useful for running test suites when I needed to include postgresql/redis/what have you into the environment.

EG is about integrating the entire lifecycle from local development to deploying into production to managing SOP for deployed environments using virtualization.

You can think of it as codespaces/temporal combined without ever needing to touch a cloud provided system.

My main problem with those systems is essentially: not local first, priced for large companies, and usually require a team of engineers to maintain for the company. essentially I find those to be the antipatterns in software today. (I dont consider EG a CI/CD tool to be honest, just a problem it handles really well)

You are mistaken on many accounts:

1. Bazel/Buck is not just for building, it is a task runner with built in distributed execution and caching. Which is, uh, what CI is.

2. You not finding it useful might be because you want to continue to use the same methods and tools yet get different results. Bazel/Buck supports Go and Rust and those just fine (JS is steadily improving as well, I know multiple public companies using it). Now, the DX may be different or non-standard but that's the point...different methods for different results. Of course there are tradeoffs to it not being the standard way to do things for a particular language!

3. You can run postgresql/redis/what have you into the environment with Bazel/Buck, all the companies using these tools do it. Almost everyone wires it up to https://testcontainers.com/, see https://www.docker.com/blog/revolutionize-your-ci-cd-pipelin....

4. For integrating the entire lifecycle, this is how it is set up at most companies...use Bazel/Buck for building the environment, use Bazel/Buck for building, use Bazel/Buck to trigger a deploy, etc. For example, see https://github.com/adobe/rules_gitops

Again, not trying to be rude, just drop info. Your list of pain points in the post is real, but only with certain / older systems or ways of using them. If you look into the newer systems, you'll see that they were largely created to solve those issues!

It's always interesting to see different tools in this space. I'm glad that people are trying to innovate on CI/CD workflows. @pandiculation, have you checked out Dagger? How does EG compare with it?

Reply on news.ycombinator.com