HNHacker News
TopNewBestAskShowJobs

crohr

331 karma · joined January 23, 2012

Maker of fine DevOps tooling since 2013.

Currently launching https://runs-on.com, 10x cheaper GitHub Action runner, on premise.

More details at https://cyrilrohr.com

submissionscomments
crohr··on Gitlab is reportedly up for sale
> self-hosting GHA Runners (plural!) is ... hard (not impossible, and quite doable with enough investment, especially if one takes the dive into the murky waters of k8s)

I’m the maintainer of RunsOn (https://github.com/runs-on/runs-on), which solves this problem with far less investment

crohr··on Faster Docker builds using a remote BuildKit instance
Another option is to simply cache layers in a fast cache close by (e.g. S3) ? Like https://runs-on.com/features/s3-cache-for-github-actions/#us...?
crohr··on Ask HN: Any CI/CD SaaS Solutions for ARM?
You could also be interested in https://runs-on.com.

Must be installed in your AWS account (1-click cloudformation template), and then you can use any instance type offered by AWS.

crohr··on Runs-on: Self hosted, on-premise, GitHub Action runners
It’s going to be easier to maintain and cheaper. Plus you get compatible images included and more flexible runner assignment. But if you’re already happy with ARC then there is no real need to switch. RunsOn is really for people who want more of a turnkey solution.
crohr··on Runs-on: Self hosted, on-premise, GitHub Action runners
The software is on-premise, as in it runs on infrastructure you control, as opposed to a SaaS. Self-hosted might be the better term, but in the context of self-hosted GitHub runners, it’s a bit of a mouthful to say self-hosted solution for self-hosted runners.
crohr··on Arm64 on GitHub Actions
RunsOn is the only on-premise solution of the bunch, so it is single tenant, and as such it cannot easily have pools of 10s of machines on standby to start within 10s. So on queue time I’m comparing Ubicloud against the other SaaS providers, and anything higher than 20s is not especially great in that case. Plus you have high variance, which I personally consider an issue.

Once this improves I’ll be sure to update it.

crohr··on Arm64 on GitHub Actions
> We were using self-hosted runners which were not as efficient as we thought so we spawned EC2 instances and connected them to the github network to be runners, but we had such an amount of issues that we wanted to go back to non-self hosted

For those hitting that same kind of issue (and you are many), you should check out my product https://runs-on.com. 1-click install, 1-click upgrades, and from 7x to 16x cheaper runners on AWS.

crohr··on Arm64 on GitHub Actions
The queue time for x64 appears to vary quite a bit still. But you are correct that it is much better than what it was a few weeks back, when the analysis was written (it was not unusual to see 60s+ spikes).

Also note that I specifically mentioned that "all third-parties are good on that front, expected [sic] Ubicloud (but that may change)."

Also agree that now that you removed the outdated CPUs (was still active end of May), the analysis should mention that you have OK CPUs by now. I will fix it.

For providers that are on this page, do not hesitate to write to me if you find the analysis outdated for your specific service.

[edit] page is now up to date with new analysis

crohr··on Arm64 on GitHub Actions
Also a bit weird that they now seem to delegate the image building to third-party vendors (first GPU images, now ARM). Would be nice to know how they validate base images before distributing them to everyone.
crohr··on Arm64 on GitHub Actions
Much welcome addition, although not available (yet) on free plans. I benchmarked them and they are ~15-20% slower than AWS Graviton processors: https://runs-on.com/benchmarks/github-actions-runners/#arm64...

But at 37% cheaper than x64 it's quite a better deal than what the outdated x64 CPUs.

crohr··on Making EC2 boot time faster
m7i CPU is in the same ballpark figure than m7a (https://runs-on.com/benchmarks/aws-ec2-instances/). When you look at the interruption percentage for m7a I think m7i (not m7i-flex if you don't want burstable instances) is probably the better choice. But I suppose it depends on availability in their specific zones.
crohr··on Making EC2 boot time faster
> while we can boot the Actions runner within 5 seconds of a job starting, it can take GitHub 10+ seconds to actually deliver that job to the runner

This. I went the same route with regards to boot time optimisations for [1] (cleaning up the AMI, cloud-init, etc.), and can boot a VM from cold in 15s (I can't rely on prewarming pools of machines -- even stopped -- since RunsOn doesn't share machines with multiple clients and this would not make sense economically).

But the time taken by the official runner binary to load and then get assigned a job by GitHub always takes around 8s, which is more than half of the VM boot time :( At some point it would be great if GitHub could give us a leaner runner binary with less legacy stuff, and tailored for ephemeral runners (that, or reverse-engineer the protocol).

[1] https://runs-on.com

crohr··on We managed to speed up our CI to save 168 days of execution time per month
Any time I see engineering time spent on splitting/sharding test suites, I can't help but wonder if access to a single beefier runner (e.g. 64+ cpus) would have alleviated all that work. Also always find the duplicated setup time a bit wasteful on resources.
crohr··on Show HN: Managed GitHub Actions Runners for AWS
Oh yes, it can't work with GHA hosted runners otherwise you'll pay egress fees. From your first post I was assuming you were starting from the point of view that you would be running your own runners already.

It does work as advertised, try it :) And yes RunsOn is a direct competitor to the 5 YCombinator-funded companies operating in this space (Ubicloud, Warpbuild, Buildjet, Blacksmith, Depot).

crohr··on Show HN: Managed GitHub Actions Runners for AWS
What you are looking for is a local S3 cache, which buildx supports as a backend. Just make sure you have an S3 gateway connected to your VPC (and that your S3 bucket is in the same region than your runners!) and enjoy free bandwidth, unlimited cache size, and crazy fast network throughput.

https://runs-on.com/reference/caching/ https://runs-on.com/features/s3-cache-for-github-actions/

crohr··on Show HN: Managed GitHub Actions Runners for AWS
I believe the solution is to decentralise, i.e. let the customer run the machines in their own AWS account (what I'm doing with RunsOn, link in bio if interested).

It is very hard for a single player to get favourable treatment from Azure / AWS / GCP to handle many thousands of jobs every day / hour.

I wish Depot all the luck, I think they've done good work wrt caching.

crohr··on Show HN: Managed GitHub Actions Runners for AWS
Yes, BuildJet runs from Hetzner - https://runs-on.com/reference/benchmarks-gha-providers/
crohr··on AWS Makes Cloud Formation Stack Creation Up to 40% Faster
If you’d like to try something else, RunsOn supports any type of account (https://runs-on.com). No MacOS though.
crohr··on AWS Makes Cloud Formation Stack Creation Up to 40% Faster
Great to see some improvements on that front. CloudFormation is perfect for point-and-click install of third-party software (like mine [1]), but the slowness doesn't make for a great first impression when setting everything up.

Really good thing is the declarative nature of it. When you release upgrades to AWS accounts where you don't have any control, this makes everything safer (and simpler to manage).

[1] https://runs-on.com/guides/install/

crohr··on What's the fastest EC2 instance CPU?
You're right, just added m5zn. Base frequency is 3.8Ghz and turbo boost 4.5Ghz, but since it is an outdated CPU, they are average at best.

I'll also benchmark the z1d.

crohr··on Show HN: Open-source x64 and Arm GitHub runners
I don't understand what makes the pricing a non starter. You mean the license pricing? If so, 300€/year for something that will save you thousands doesn't seem too expensive to me. If you're talking about the instances, it is 10x cheaper than comparable GitHub runners (i.e. same as ubicloud), so not sure what's wrong here :)
crohr··on Show HN: Open-source x64 and Arm GitHub runners
I love to see examples in the wild of repos having switched to a non-official GitHub Action runner. I'm slowly preparing a collection of repo timings comparing GitHub vs Buildjet/Warpbuild/Ubicloud vs my own solution RunsOn.

In your case your workflow can run in less than 5 minutes on AWS ephemeral machines, for the same price as ubicloud: https://github.com/runs-on/arroyo/actions/runs/7723361513/jo...

crohr··on Show HN: Open-source x64 and Arm GitHub runners
Just a catch with Hetzner vs AWS: pricing at AWS is per-second while Hetzner is per-hour, which is (very) inconvenient if you're launching ephemeral runners.
crohr··on Show HN: Open-source x64 and Arm GitHub runners
One of the goal of my own hosted runner project [1] is to display the carbon cost and money cost after each run. Too many people are oblivious of the costs and I think as an industry it would be great to at least get the data points.

[1]: https://runs-on.com

crohr··on Show HN: Open-source x64 and Arm GitHub runners
You can reach 10x cheaper with just AWS, especially on the larger runner sizes. That is even including disk costs.

I put a calculator for RunsOn at https://runs-on.com/calculator/

crohr··on Show HN: Open-source x64 and Arm GitHub runners
Yes, I wonder when/if GitHub decides to drop prices at some point.

But in the meantime I’ve recently released RunsOn [1] with the same promise (10x cheaper, faster) but the whole thing runs in your AWS account.

[1] https://runs-on.com

crohr··on A supply chain attack on PyTorch
I've just made runs-on [1] for that purpose: self-hosted, ephemeral runners for GitHub Action workflows. Long-running self-hosted runners are simply too risky if your project is public.

[1]: https://runs-on.com

crohr··on runs-on: [cheap,fast,self-hosted] runners for GitHub Action, in your AWS account
Hello HN! I'm showing you RunsOn, a new service to launch self-hosted runners in your own AWS account, without the usual SaaS middleman.

RunsOn is installed in one click with a CloudFormation template, which deploys a private GitHub App for the organization of your choice (even personal).

The App then receives workflow job events from GitHub, and takes care of spawning EC2 instances to act as your GitHub runners.

In your workflows, you then have to do this simple change:

- runs-on: ubuntu-latest

+ runs-on: runs-on,runner=16cpu-linux

The README has a lot more details as to how you can customize the runs-on label.

Some key advantages:

* up to 10x cheaper than GitHub, and 4x cheaper than the competition.

* 1-1 workflow compatibility for your existing workflows, since we maintain replicas of the official runner images.

* x64 and arm64 architecture are supported.

* bring your own AMIs, or customize the default ones with preinstall scripts.

* your code stays in your own infrastructure. * we take extra precaution to ensure no runner is left dangling, thanks to AWS TerminateOnShutdown feature and watchdog scripts.

I also made sure that the stack is easy to troubleshoot:

* any problem when starting runners is automatically surfaced in a special management issue.

* daily cost usage is also displayed in the management issue.

* SSH access to the runners if needed.

It has been a lot of joy creating this tool, and most companies having a bill of a few hundred dollars on GitHub will quickly see the ROI, especially if you need higher-spec runners.

Disclaimer: the tool is free for non-commercial usage, and comes with a flat license fee for commercial usage.

crohr··on Show HN: WarpBuild – x86-64 and arm GitHub Action runners for 30% faster builds
Exactly my experience as well: https://x.com/crohr/status/1732442731715113374

In the tests with my GitHub Action [1] that spawns ephemeral runners for any workflow, I found BuildJet bandwidth speed 10 to 20 times slower than machines at AWS.

[1]: https://github.com/runs-on/action

crohr··on Show HN: WarpBuild – x86-64 and arm GitHub Action runners for 30% faster builds
If you need machines with high number of CPUs, you can check https://github.com/runs-on/action which allows you to select any EC2 instance type as ephemeral runner. Plus it’s the cheapest on the market, open source, and not a SaaS
← PreviousPage 2 of 3Next →