Hacker News Reader

Comment by joshstrange | Hacker News Reader

GitHub’s runners are a joke. Switching to WarpBuild was the best decision I made for my CI/CD. I halved my build time and got cheaper minutes.

GH’s macOS runners particularly were complete trash.

View on HN

We're a small team (four engineers) and we purchased four Dell mini PCs to toss into our small office space. The break even compared to our AWS hosted runners (which were much faster and cheaper than the GH ones) was like four or five months, plus they cut the time again by over half. We have to deal with them if something weird happens, but it's been such a low time investment that it has been such a great choice.

Plus, who doesn't love having some new hardware to play with?

Why 4 Dell Minis? You don’t need a monitor for CI/CD, and why 4? Is it for redundancy? Performance?

We are interested in doing the same setup.

Not OP, but I went with MOREFINE AMD Ryzen mini PCs. They can be configured with a ton of CPU/RAM (including ECC). They're really compact, support 2.5 Gb ethernet, and are great to work with. A comparable Dell workstation with ECC RAM would've been significantly more expensive, and you have to deal with their business folks to purchase them (or at least I couldn't figure out how to buy them direct).

Wasn't my choice, but one wasn't enough so we threw a few more at it. So mostly for performance, but the redundancy is an added benefit. They were a pretty decent space, power/heat, performance, and noise ratio for us.

Our office is a single room with six desks, so space, noise, and heat are a larger consideration for us. I use a mini PC at home for a personal server as well, but that's just because used enterprise equipment is often a great deal.

> We have to deal with them if something weird happens

Check back in three years once you've had to deal with OS upgrades, power outages, running out of disk space, and one of them failing.

Three years sounds like a reasonably good return on investment. At that point the teams either big enough to justify paying $$ for a hosted solution, or still cheaper to buy dedicated hardware.

You can also just rent dedicated servers from hetzner or ovh for far cheaper than AWS instances with more horsepower. More expensive than a dell mini in a closet. But you get datacenter Internet and power backup.

> justify paying $$ for a hosted solution, or still cheaper to buy dedicated hardware.

Always cheaper if you have at least some demand. For the lazy fix even just get a few mac mini and it'd work fine.

It's a CI runner my dude.

You just obliterate and reinstall the base system in what 10 minutes?

Are any of those problems challenging to deal with? Even if it’s a few expensive man hours the ROI is still quite short.

if the break even was 4-5 months, it means compared to going to cloud vendors, in 36 months, the self hosted solution would have saved 7.2 times the cost!!

Even if within 3 years there needs to be OS upgrades, you can just e-cycle the old machines and buy brand new minipc's (with added benefit of increased performance) every time you need to upgrade, and you'd still be ahead!!

We're about eight months in at the moment, so here are some current stats:

- No OS upgrades (running Ubuntu 22 LTS), but regular update

- We've had a power outage, but not during the day, so when they rebooted, they went right back to action. Same issue with a network outage in the office.

- We have hit disk space issues due to not clearing all the data we needed to. A quick SSH and an rm or two did the job, plus another 30 minutes to an hour to update a cron to remove that portion of the cache

I see what you mean for sure, and there will undoubtedly be more things that come up at some point, but with the savings, it would take quite a few engineering hours per month to lose the savings at this point.

And I won't lie, when something goes wrong, it's a nice change of pace to deal with a server instead of code once and a while. Can't actually use that for calculations, but I don't hate when it happens so far.

When a better ROI of 4-5 months, you could just buy a new one when any of those things happen, and still be net positive.

> Check back in three years once you've had to deal with...

This being routine engineering, I can tell you right now how it will go. Let's start!

> OS upgrades

Distinction needed: update/upgrade. Entirely automatable, some may say table stakes. Decide to update in this job or another.

Upgrades are a common problem for either hosting world. Can my software work in this new environment? Hence, CI.

> power outages

Set the firmware to return to last power state. Also buy a couple UPS' and a PDU. Fixed. A host and networks between going away isn't so easy.

> Running out of disk space

How is this unique for them to solve? Hosted workers/cloud servers have finite disks too.

Clean your workspaces in your pipelines. That's all disposing of the hosted one does.

> and one of them failing.

Someone else in the thread said it best, these are CI runners man. Capacity is down slightly. Oh no. Several cheap copies were bought to address this.

The marketing was so successful that we have to debate that yes, there is value in owning things you need/use.

Or just run your own runners on-premise. You'd be amazed at how fast builds can be with OTS hardware, or even something like a laptop or Mac mini.

I considered this for my personal business but the minutes are stupid cheap and to pay off a Mac mini it would take me many, many years to break even. For a company with 10's-100's-1000's+ employees writing code and needing CI/CD the math will be different. I came close to buying a Mac Mini anyways, costs be damned, to get away from GH's slow runners.

> minutes are stupid cheap

At smaller scales, as long as your builds aren't stupud-slow, cloud providers should be cheaper just because they can utilize their hardware more (10x?) efficiently.

Totally agree, and I called out that my scale is small so cloud makes a lot more financial sense for me (especially as a solo founder, I don't want to manage more hardware than I have to already).

I understand the Mac builds, but the entire appeal of GitHub runners is it's literally built in. I can just add a YAML file, and we have CICD!

I get that, but you can run GH runners on your own hardware or someone else's. Switching to WarpBuild was literally as easy as linking my GH account and changing the running image from `macOS-latest` to `warp-macos-14-arm64-6x` and everything worked. Secrets and everything.

> Secrets and everything.

I guess you had to import the secrets, too...

Ditto. I came for the cheaper Apple Silicon runners (before GitHub Actions had them available for free) and stayed for the faster runs across the board. I cut a 24 minute build down to 10 minutes just by switching over to WarpBuild runners. Not a shill, just a happy customer.

And I a happy provider - thank you!

Can't agree more with this, I recently migrated my company to use WarpBuild and our CI got 50% faster, not even kidding. We were using self-hosted runners which were not as efficient as we thought so we spawned EC2 instances and connected them to the github network to be runners, but we had such an amount of issues that we wanted to go back to non-self hosted. At the end a tech lead of the company suggested using Warp and we could not be happier with the results.

Also the CEO Surya is such a great guy, we had a small networking issue that lasted for about two hours and they gave us credits for all the runners affected with that, and their customer support is blazingly fast.

Thank you so much for the kind words!

> We were using self-hosted runners which were not as efficient as we thought so we spawned EC2 instances and connected them to the github network to be runners, but we had such an amount of issues that we wanted to go back to non-self hosted

For those hitting that same kind of issue (and you are many), you should check out my product https://runs-on.com. 1-click install, 1-click upgrades, and from 7x to 16x cheaper runners on AWS.

Brilliant idea!! I just dug through your docs. You significantly undercut GitHub on runner pricing, using your customers’ infrastructure for both runners and their orchestration, which is managed by your software, some cloudformation, and amis that you maintain and license in return for a small slice of that savings. I love it, and if our cloudops org ever threatens to charge back CI to our budget I’ll be a customer.

I was deploying to CloudFormation and I thought that was the slow part but my "build" times went from 20min to 10min. My iOS builds also were halved.

GitHub Actions is a joke. It baffles me that they've been breaking file permissions on upload for years and they haven't fixed it yet. God forbid you want to ship an executable in a tarball or save it for a later stage of your build.

Different needs, different tools! I work on a open-source project, and not having to pay for CI/CD with a "simple" yml file configuration is simpler and easy to manage.

It's only free if your time is worthless :P every couple weeks I have to debug something that I can't replicate locally because I don't have local CI

That's the other plus with something like WarpBuild, I haven't used it yet but you can pause the build and SSH in:

https://github.com/WarpBuilds/action-debugger

Have you tried using act for that? https://github.com/nektos/act Has saved me a bunch of CI debug pain.

You can try FlyCI Wingman - an AI agent that analyzes and fixes your failing builds in GitHub actions. We're about to release it tomorrow, so feel free to subscribe and be among the first to try: https://www.flyci.net/ .

If you are open source and getting free minutes then it really doesn't matter how long builds take.

I'm still using GH Actions, just not their minutes.

How do you handle extremely large repos? Can these be cached on machines, per-customer?

Assuming the question was reg. WarpBuild - we provide blazing fast caches with unlimited sizes. These still need to be downloaded to machines and setup on each workflow job.

We don't (yet) have support for pre-pulled repos on runners per customer. Are the large repo sizes despite fetching only the refs required for the specific context [1][2] which can help speed things up significantly.

[1] https://www.atlassian.com/git/tutorials/big-repositories [2] https://stackoverflow.com/questions/48072080/how-to-manage-l...

There is a lot of inefficiency in CI today and we're out to solve it.

Thanks for the WarpBuild love!

How do they compare to Ubicloud?

https://www.ubicloud.com/

I think I skipped over evaluating ubicloud because they don't have macOS runners. There was another place I liked but I needed macOS runners they didn't have them either.

(WarpBuild founder here) We have had a lot of users evaluate multiple tools including ubicloud but chose us. Typically, they've mentioned our runners being faster, our caching is very useful, and queueing delays being lower.

However, my last anecdata was from a few weeks ago. You can try us out and take a call for yourself.

Our focus is on CI runners and efficiency and I think that gives us an edge on the devex. Ubicloud is definitely cheaper though.

Reply on news.ycombinator.com