BYOC Anywhere: The Spectrum of Bring Your Own Cloud Deployments | Hacker News Reader

BYOC Anywhere: The Spectrum of Bring Your Own Cloud Deployments

omnistrate.com

80 pointskkgupta16 comments

Actual title is "BYOC Anywhere: The Spectrum of Bring Your Own Cloud Deployments"

To me, BYOC will always mean "bring your own computer", which is amusing in this context.

apparently computer + k8s connected to internet is considered cloud now

"BYOC" deployment plans often require more than just k8s. "S3-compatible object store" is the most common one, IME. Or other sorts of scalable storage systems.

("Why does the vendor not just include that in the k8s they send you?" Usually because it's easier for them to leave everything about physical storage to the env maintainers.)

I thought that fell under BYOD (device) :)

In my mind, BYOD is like being able to use my own iPhone or laptop to connect to some corporate network. I think of this "device" as a client and I think of byoc as being able to run some application or constellation of applications on my own device or in a vendor neutral way on some cloud like a podman compose or something similar... Like not dependant on a particular vendor's "native" capabilities

BOYD is corporate to me, BYOC LANs were common here

Nice write up. I have been working on an airgapped k8s-based, robotics simulation platform for awhile and this is one of the most accurate and coherent writings I've found on the subject.

I'll definitely be sharing this with my team.

This is a great document for understanding all of the terminology/considerations relative to cloud computing buyers -- a must read for cloud computing buyers, salespeople, or really, anyone else in that space...

I work on a pretty large BYOC deployment that we market as such, but is squarely in the "self-hosted" / on-prem model. We don't maintain any connection back to our environment and updates are driven by customers. It's pretty difficult for all the reasons you can likely think of.

This space is quickly becoming a bit saturated, as these "BYOC enablement" tools and platforms are starting to crop up, and I can see the value in them, but my experience doesn't always line up with it. I'm actually really interested in hearing about a large enterprise running vendor code that would even entertain the idea the vendor could automatically update the entire deployment without their direct involvement. Our platform is quite load bearing for our largest customers, and in some cases directly in their revenue stream. There is no way they would ever let us push updates without going through their own internal controls of validation, testing, and sign-off. One could definitely argue these controls aren't entirely founded in reality, but corporate reality isn't always the reality you and I might share.

Besides the issue of automatic updates, we struggle with wild configuration differences that need pretty direct involvement with the customer. Security groups in AWS are just one example of this: some customers allow all internal VPC traffic between services, and others are quite strict in that every service needs a perfectly scoped security group with inbound/outbound rules. These have to be provisioned with their own tools built by their own teams under their own version control. We just ask for a configuration file in a well known location that defines these IDs we can reference in the CloudFormation.

So knowing that we essentially have to operate in the self-hosted model, CloudFormation becomes our mechanism for defining consistent deployments. For us to switch over to something like k8s just to fit into the model a BYOC-enablement tool like this wouldn't be worth it.

I think the most important aspect here though, and the most subtle one, is the issue of deployment and operational complexity. For BYOC to work well, the vendor really should be designing for it from the beginning. Taking an existing multi-tenant SaaS architecture and shoving it into this model, even with the ideal BYOC vendor-maintains-control-plane scenario is no small task. Even when you're talking about large enterprise BYOC deployments, you may only have a few thousand daily active users, which is dramatically less than what most SaaS vendors design for when operating in their own multi-tenant environments. ("Web scale" and all that.) At that point, I think engineering teams need to take a serious look at their architecture, and realize just how far they can get with fewer services, fewer machines, and fewer "event driven" architectural decisions. The deployment model _should_ be a forcing function for simpler architecture and system design, but I realize that's not a given. I think what I'm trying to say is that, knowing BYOC done well requires designing for it from the beginning, a well understood daily load profile and operational context kind of eliminates (or at least dramatically reduces) the need for tools like this, and that self-hosting isn't the worst possible outcome.

I agree, I work on an "airgapped" BYOC system as well, that runs on data lake providers mostly, and it's a bit of a nightmare because each customer has their own labyrinthine network setup, their own way of ingesting data (as much as we try to be prescriptive, some of them just insist on doing it their way), vastly differing scales and shapes, and sometimes their own secure communication tooling to even read the emails describing the problems they have. It's basically shrinkwrap software.

We're moving away from it as much as possible lately for all but the biggest customers with the strongest isolation needs.

I am also struggling with coworkers thinking they can cowboy code some LLM written features into these codebases recently, not understanding the level of human verified testing that needs to catch all the manifold edge cases and weird decisions in their usage. An update that causes crashes is much harder to catch when you can't do slick blue-green deploys and similar things to roll out updates.

One nuance on the air-gapped / shrinkwrap end of the spectrum that I was thinking about afterward is that there is a difference between “no connection back to the vendor” and “just ship binaries and scripts.” One can also imagine shipping a local control plane with the product so provisioning, health checks, upgrades, rollback, etc. still follow a consistent operational model entirely inside the customer boundary. There are several examples in the industry that take this route as well. Obviously that doesn’t remove the enterprise change-control problem — updates to that local control plane still have to go through the customer’s own approval/import process. It just gives you a bit more operational consistency than traditional shrinkwrap software.

The operational challenges that you highlighted with BYOC makes sense and they need to be addressed.

- On upgrades, agree that “vendor managed” shouldn’t mean “the vendor can push arbitrary code whenever it wants” especially with the supply chain risks. Our experience is that vendor should make the new upgrade available and notify the customer, customer can run its change-control pipeline and approve the version, the control plane automates rollout/health-check/rollback, etc. To your point, automation only removes the operational toil but not the governance.

- Same with the networking, IAM, etc. A BYOC platform can’t always assume it owns the customer environment or one size fit all. Some customers will let you provision everything; others will insist networking/security infrastructure comes from their own Terraform/CloudFormation pipelines and give you IDs or interfaces to consume. I think a good BYOC model has to make that boundary explicit.

- Also, rewriting your app onto Kubernetes just to fit a BYOC platform would be backwards. Agree that the BYOC tooling should adapt to existing application architecture, not dictate it.

I am not sure if simplifying the application removes the need for this tooling as much of the complexity lies with establishing the enterprise trust boundary across accounts with things like identity, networking, permissions, artifact approval, upgrades, drift, auditability, governance controls, and eventually exit. This website tries to cover it in detail: https://byocanywhere.org/, WDYT?

Finally, I feel that there is a value in shrinkwrap software deployment model especially if one has the requirement of a relatively small number of very large, disconnected installations, and high-touch support. BYOC model make more sense if you want to scale or customers like to preserve customer control and managed experience like Databricks, Anyscale, Snowflake..

> Our experience is that vendor should make the new upgrade available and notify the customer, customer can run its change-control pipeline and approve the version, the control plane automates rollout/health-check/rollback, etc.

As it stands today, we have an AWS-only product (for better or worse) so we can accomplish this already with CloudFormation. It's not the best "control plane" as far as updates and rollbacks go, but it works pretty well for our needs. Obviously, trying to do the same across multiple cloud providers, would mean bringing in a different tool. At that point, the vendor's architecture would need to be neutral as well.

> I am not sure if simplifying the application removes the need for this tooling as much of the complexity lies with establishing the enterprise trust boundary across accounts with things like identity, networking, permissions, artifact approval, upgrades, drift, auditability, governance controls, and eventually exit. This website tries to cover it in detail: https://byocanywhere.org/, WDYT?

I think that website is a solid resource. Those are certainly important issues to solve that all surround the actual vendor code itself, and I can see value in making an attempt to solve it. It's just my experience that every customer wants to do it differently. I'm just not sure how much slower and painful our contract negotiation would have been if we had to also sell them on a particular process other than "you provision these things with your own tools, and we'll hand you CloudFormation templates". Maybe other companies offering BYOC are better at selling the whole package, not just the "platform" itself.

> I'm actually really interested in hearing about a large enterprise running vendor code that would even entertain the idea the vendor could automatically update the entire deployment without their direct involvement.

A better word than "involvement" is "knowledge." We operate BYOC installations of managed ClickHouse and allow customers to pick upgrade windows. It's not that different from doing upgrades yourself. It's also important to remember that SaaS products upgrade without user intervention and often without user knowledge. That's one of the characteristics of the SaaS model.

> I think the most important aspect here though, and the most subtle one, is the issue of deployment and operational complexity. For BYOC to work well, the vendor really should be designing for it from the beginning.

This is definitely correct. We made a design decision early on that environments for managed databases (i.e., the data plane) are dedicated Kubernetes clusters for each tenant. Data plane services fit in the single K8s cluster and can be installed in 20 minutes or less over the network.

The Omnistrate article makes a great point that there are different flavors of BYOC depending on how much control users want. We have two variants: (a) fully managed stack in its own VPC where we install and run everything and (b) managed databases on user-supplied Kubernetes.

Great write up! This is so critical to get workloads to production

Reply on news.ycombinator.com