HNHacker News
TopNewBestAskShowJobs

suryao

381 karma · joined May 29, 2019

Founder, WarpBuild.com - fast, secure Github actions runners that speed up builds by 30+% and at half the cost. Get started in 2 mins.
submissionscomments
suryao··on Docker limits unauthenticated pulls to 10/HR/IP from Docker Hub, from March 1
This sucks for individuals and open source. For folks that have a heavy reliance on dockerhub, here are some things that may help (not all are applicable to all use cases):

1. Setup a pull through mirror. Google Artifact Registry has decent limits and good coverage for public images. This requires just one config change and can be very useful to mitigate rate limits if you're using popular images cached in GAR.[1]

2. Setup a private pull through image registry for private images. This will require renaming all the images in your build and deployment scripts and can get very cumbersome.

3. Get your IPs allowlisted by Docker, especially if you can't have docker auth on the servers. The pricing for this can be very high. Rough numbers: $20,000/year for 5 IPs and usually go upwards of $50k/year.

4. Setup a transparent docker hub mirror. This is great because no changes need to be made to pipelines except one minor config change (similar to 1). We wrote a blog about how this can be done using the official docker registry image and AWS.[2] It is very important to NOT use the official docker registry image [3] as that itself can get throttled and lead to hairy issues. Host your own fork of the registry image and use that instead.

We spent a lot of time researching this for certain use cases while building infrastructure for serving Github actions at WarpBuild.

Hope this helps.

[1] https://cloud.google.com/artifact-registry/docs/pull-cached-...

[2] https://www.warpbuild.com/blog/docker-mirror-setup

[3] https://hub.docker.com/_/registry

suryao··on Show HN: Lume – OS lightweight CLI for MacOS and Linux VMs on Apple Silicon
Check out my project, WarpBuild [1], if you want managed Mac VMs for CI/CD purposes. It plugs in nicely to the GitHub actions ecosystem and provides machines on demand.

[1] https://warpbuild.com

suryao··on GitHub Linux ARM64 hosted runners now available for free in public repositories
hey - https://www.warpbuild.com is the website the pricing is https://www.warpbuild.com/pricing
suryao··on GitHub Linux ARM64 hosted runners now available for free in public repositories
Ha,forgot to add for private repos.
suryao··on GitHub Linux ARM64 hosted runners now available for free in public repositories
For cheaper (for private repos) and faster arm64 runners, check out what we're making at WarpBuild.

We also support spinning up self-hosted runners on your AWS/gcp/azure in just a couple of clicks.

suryao··on Comparing AWS S3 with Cloudflare R2: Price, Performance and User Experience
Great article. Do you have throughput comparisons? I've found r2 to be highly variable in throughput, especially with concurrent downloads. s3 feels very consistent, but I haven't measured the difference.
suryao··on Guide to Self-Host GitHub Actions Runners on EKS
I put together a detailed write-up on setting up self-hosted GitHub Actions Runners using ARC (Actions Runner Controller) on AWS using EKS.

This includes terraform code for provisioning the infrastructure, and helm configurations for Karpenter v1.0 setup. We also ran a couple of variants in configuration for cost and performance comparison using Karpenter for autoscaling and other best practices.

Using a couple of PostHog's OSS repo workflows as for testing, the variations of config were basically allowing Karpenter to pick runners of arbitrary sizes from the same instance family and figure out the scaling, vs forcing one node per job.

The most interesting part from the post is added below.

## Performance and Scalability

All the jobs are run on the same underlying CPU family (m7a) and request the same amount of resources (vcpu and memory).

ARC runners with varied node sizes exhibited slower performance primarily because multiple runners shared disk and network resources on the same node, causing bottlenecks despite larger node sizes.

To address these bottlenecks, we tested a 1 Job Per Node configuration with ARC, where each job ran on its own node. This approach significantly improved performance. However, it introduced higher job start delays due to the time required to provision new nodes.

Job start delays are directly influenced by the time needed to provision a new node and pull the container image.

Total Jobs: 960 ARC (Varied Node Sizes): $45.88 ARC (1 Job Per Node): $42.60

What are some other optimizations that can be done? Are there other considerations that could be added to extend the post?

Let me know what you think.

Full post: https://www.warpbuild.com/blog/setup-actions-runner-controll... Code: https://github.com/WarpBuilds/github-arc-setup

suryao··on Your CI pipeline isn't ready for AI
This is absolutely true. We run fairly heavy duty CI infra and I can say with confidence that this is one of the biggest factors impacting CI runner performance, almost equal in weight to raw CPU perf.
suryao··on Ask HN: Is GitHub actions down?
If it needs to be asked, historically the answer has been a resounding Yes.
suryao··on Show HN: GitHub Cost Visibility and Optimization
This is very useful. How do you get the self-hosted runner costs - directly with the aws/gcp/azure integration or something else?
suryao··on Ask HN: How much does CI/CD cost your team/company?
It is definitely a high-spend area. It is not uncommon for teams that are <50 people to spend ~$2-5000/month on CI, sometimes more.

There are repos we've seen that are heavily sharded for testing and trigger ~50-200 separate jobs per commit, each taking a few minutes to run.

CI costs are only going to increase with each engineer putting out 10x code and autogenerated tests in the next few years.

aside: we're tackling this problem as a company. I've been considering putting out an analysis of this data from our customer set - your post might just trigger me to do it.

suryao··on Container Layer Caching for 2-5x faster builds
Thanks for posting about our recent launch.
suryao··on Gitlab is reportedly up for sale
Thought I'd plug my product because of the mention of actions-runner-controller on k8s. I wouldn't wish that on my worst enemy :)

It is a terrible experience riddled with many gotchas. I'm making WarpBuild to provide runners (cloud and BYOC on user's aws account) to provide more powerful, faster, and overall much better experience. Try us out - you're guaranteed to save both money and time.

suryao··on Faster Docker builds using a remote BuildKit instance
incremental CI is absolutely the way to go
suryao··on Faster Docker builds using a remote BuildKit instance
This is pretty cool - provides a good speed up for container builds. The couple of beefy instances can set you back $200-1000 a month on aws apart from the regular github action runner costs and it only goes up from there. We have a way around that plus effective scaling for multiple parallel builds with WarpBuild.

As a side note: In my time running a CI infra co, we see that a majority of the workflow time for large teams comes from tests - which can have over 200 shards in some cases.

suryao··on Show HN: 10x cheaper GitHub Actions on your AWS account
Any such error is a bug. We do have tons of ruby users running 1000s of jobs each hour but maybe we are hitting a corner case with you, I'm sorry that that happened and I'll look into it.

Switching from github actions default runners is exactly a one line change.

Migrating from `actions-runner-controller` with k8s or self-managed VMs could be different based on specific customizations in place. However, we have import flows for the former to directly import the custom containers used in k8s, and an AMI import is coming soon.

There shouldn't be any stack specific behavior, except with caches. We have introduced fast custom caching actions including stack specific instructions in docs [1].

[1] https://docs.warpbuild.com/cache/quickstart#ruby

hth.

suryao··on Show HN: 10x cheaper GitHub Actions on your AWS account
The MacOS runners are in our cloud (not user's AWS) and are usage based. Users pay only for the actual minutes of usage.
suryao··on Show HN: 10x cheaper GitHub Actions on your AWS account
Yes, but likely in ~1.5 months.
suryao··on Show HN: 10x cheaper GitHub Actions on your AWS account
Thanks for being patient with us through our teething issues. I think you'll like the stuff in our pipeline :-)
suryao··on Ask HN: Any CI/CD SaaS Solutions for ARM?
We faced that as well and that's one of the reasons for the diversification across clouds
suryao··on Ask HN: Any CI/CD SaaS Solutions for ARM?
It's AWS mostly though we do have multi cloud presence that we're expanding in the next couple of months
suryao··on Ask HN: Any CI/CD SaaS Solutions for ARM?
Check out what we're building at WarpBuild.

We provide fast compute for various architectures and OSes.

suryao··on GitHub Actions Outage
Can confirm that self hosted runners are seeing issues at WarpBuild.

Just another Thursday :shrug:

https://warpbuild.instatus.com/

suryao··on Arm64 on GitHub Actions
Not at this time. We're a small team trying to build a sustainably growing business with server bills to pay.
suryao··on Arm64 on GitHub Actions
And I a happy provider - thank you!
suryao··on Arm64 on GitHub Actions
Assuming the question was reg. WarpBuild - we provide blazing fast caches with unlimited sizes. These still need to be downloaded to machines and setup on each workflow job.

We don't (yet) have support for pre-pulled repos on runners per customer. Are the large repo sizes despite fetching only the refs required for the specific context [1][2] which can help speed things up significantly.

[1] https://www.atlassian.com/git/tutorials/big-repositories [2] https://stackoverflow.com/questions/48072080/how-to-manage-l...

suryao··on Arm64 on GitHub Actions
(WarpBuild founder here) We have had a lot of users evaluate multiple tools including ubicloud but chose us. Typically, they've mentioned our runners being faster, our caching is very useful, and queueing delays being lower.

However, my last anecdata was from a few weeks ago. You can try us out and take a call for yourself.

Our focus is on CI runners and efficiency and I think that gives us an edge on the devex. Ubicloud is definitely cheaper though.

suryao··on Arm64 on GitHub Actions
(founder of WarpBuild - we offer hosted GHA runners) This is a common issue. A common pain point we see with users approaching some scale with self-hosting on k8s is that the k8s node autoscaling can become inefficient because of spiky loads.

We have a lot of users migrating off self-hosted setups using `actions-runner-controller` to ours because of this. Essentially, not having to deal with bin-packing is more efficient and concurrency, uptime guarantees are nice.

suryao··on Arm64 on GitHub Actions
Thank you so much for the kind words!
suryao··on Arm64 on GitHub Actions
I'm so glad that GitHub finally offers this.

They're just more efficient for certain kinds of workflows - both cheaper and faster.

If you're building for arm64 targets or android, this is a no brainer because it sidesteps the emulation requirements.

We've been offering arm64 runners for a few months now (up to 32 vcpu) with WarpBuild and our users love it.

In general, there are lots of inefficiencies in CI starting from the lack of visibility, debugging ease, performance, runner config customization, persistence, etc. We're out to solve it and make the world's fastest CI cloud ecosystem on top of existing CI providers with WarpBuild.

← PreviousPage 2 of 5Next →