Karmada: Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration

No questions on the technical implementation, this looks impressive. Nevertheless, I'm getting more and more convinced that the whole multi-cloud topic is snake oil, an engineering self-purpose and basically the ORM discussion of the '00 years reloaded.

When you're betting on any of the large, public clouds, what you're paying extra on top compared to colocation is basically renting an insanely large and experienced operations team that's solely incentivized by keeping the services running and connected. I'm not saying the large players will never fail, but at least they are designed and built to that standard at any single point of contact.

Even at a _single_ public cloud, bugs and misconfigurations far outweigh any hardware failure. I've seen routing configurations or the Kubernetes backplane fail on GCP that took down my services, but never ever even a whole Availability Zone.

The farther you zoom out, the larger the probability of misconfiguration and failure by excessive technological complexity compared to individual server or service failure.

I did myself a favor once and wondered what's the actual cost if our site goes down and for how long. Surely it's high, you might be surprised to actually compare that against what you're paying for in risk management.

With that number in hand, I decided for example that the risk and adjusted cost of whole AZ failure is negligible compared to what we were paying for in intra-AZ traffic and happily put all of our production network back into one AZ (backups excluded of course).

Also, on the cost side, you'll usually get larger discounts and massively save on traffic costs if you just bet on a single cloud that should far outweigh any savings you might get by the leverage of avoiding lock-in, or buying individual services cheaper on specific clouds.

Then again, we're not servicing any life critical systems. Your mileage may vary.

I’ve seen different projects that aspire to this, but it may be the most complete. The rationale is simple: run workloads that can survive a public cloud failure, or local workloads that can failover to public clouds. Kubernetes promises to avoid vendor lock-in but this facilitates a level of resilience that could only otherwise happen with some hardcore network configuration and usually still accepting some single point of failure.

Is there a solution like this that also includes bare metal? I'm a Kubernetes novice but we run (large) Kubernetes dev/test clusters on baremetal (because it's vastly cheaper than the cloud providers). Especially the (very annoying to me) point of handling all the parts that are there with cloud providers but not necessary on baremetal. The most obvious example being load balancers/port networking: all baremetal load balancers need layer 2 or bgp ; we cannot use either so we need nodeport + Nginx ingress which sucks (nodeport opens ports which we don't want opened) but works. I'm specifically looking for a multi-cloud solution that handles this identically on all platforms including baremetal.

Like said; I'm not very experienced with Kubernetes, but reading and asking around, I have not found a good solution for it.

This is a system that is basically adding missing features as a super-set on another system. Rather than deal with one complex system, you are now dealing with a complex system within a complex system.

For this to actually pay off for you, it needs to cost less to run this super-system than the base system. Without any numbers (quality, reliability, bugs, etc) to speak to, there's no way to know if using this system will result in a better outcome. All you can do is hope that the base system is such a pain in the ass, and the super-system is so good, that somehow you come out on top.

It's inevitable that we'll see many of these as open source projects along with the cloud providers own "multi-cloud" solutions that pin the control plane to their own platforms. This is the holy grail of hybrid solutions we've been chasing for years, but whenever we arrive at a solution a replatforming effort begins. Before it was VMs, this time it's containers. Maybe, just maybe the kubernetes container wave will stick and we'll see some runtime agnostic system exist where workloads can be placed based on a set of tags e.g gdpr, eu-west-1, etc, etc.

But at the same time. What does it matter? Surely the cloud was about handing off to this magical thing somewhere far away and letting it handle the details. I think cloud got dissected down further and further into the nitty gritty details to a point where we just turned everything that was physical into an API, UI click or terraform script. It's useful for a certain operator, a certain layer in the stack, but honestly it is not useful to anyone trying to build a consumer facing product. It is a level of detail no one should have to contend with. Cloud vendor risk, is the same as datacenters of the past. There's tradeoffs that you accept in the face of the ability to move faster and letting that cloud providers handle everything. There's always the potential for black swan events but the optimisation for that 0.001% leads to overly complex solutions for the sake of it.

I really hope that in a few years we are not talking about this stuff. I really hope cloud is reimagined as an ever evolving network, where you build programmatically for a local experience and it gets replicated everywhere without having to think about the details. Hopes and dreams and who am I kidding, these things never happen.

How does it work with Ingress?

Is there a website other than the GitHub page? You might consider a name change before you reach 1.0, given this already exists: https://www.karmada.org/. They came up first in web search and you came up second, but they have an actual web page.

Some implementation questions. First, what is this perceived to buy on top of existing systems like Helm and Rancher that use templating engines and overrides at build and deployment time to support deploying identical services to multiple clusters possibly hosted across multiple cloud providers? You remove some of the complexity of dealing with Go templates, but add an entire new control plane and a master cluster that I'm not sure is needed if all it gives you is another level of indirection regarding where you deploy to.

Second, regarding the quick start guide, why use resets of the KUBECONFIG variable to control which cluster you're talking to? Your scripts can merge configs and set current-context so the user can just use contexts instead. Also, the KUBECONFIG variable itself supports multiple files, so you can do this without having to merge definitions into a single file if you want the control plane to be in the special /var/run path instead of the user's home. Last, you may want to change the output of the quick start to indicate the kubeconfig file generated for the host cluster is going to be in the current user's home, not necessarily root. I'm guessing whoever wrote the guide was simply running as root (probably not recommended, but a lot of people like to live dangerously) and copy/pasted out of their own terminal. Granted, they'll see the real location when they run the script anyway, but the README gave me the impression at first it will always put the file in root's home, which would clearly fail if you're not running as root. Lastly, your local-up script should support installing the control plane into different local cluster solutions instead of having a hard dependency on kind. People might be already using k3d, minikube, or even have a homelab setup with a fleet of NUCs or libvirt hosts and they don't want yet another kubernetes installation on their laptop. You might even prefer using something else for your own development. For instance, k3d does all that docker hackery to inject container ip as the api-server endpoint for you, so it supports multiple clusters on the same host out of the box without you needing to manually manipulate the kubeconfig after creating a cluster.

Also, maybe even de-emphasize multi-cloud? All the other comments so far seem to be fixating on that as a "99% of all applications don't need to be multi-cloud" bugaboo point you see all the time posts like these happen, but this is really just supporting deployments to multiple clusters with cluster-specific overrides of api object definitions. In some cases, like storage plugins, those may be specific to a cloud-vendor, so in that case you're also buying the potential for a multi-cloud deployment. But whether or not an application is multi-cloud is a separate concern on top of it being multi-cluster. Multi-cluster is required a lot more than multi-cloud, i.e. if you scale to the size where kubernetes simply doesn't support that many nodes in one cluster, or you want to easily deploy multiple versions of the same application, or you simply want to be able to orchestrate staging and production clusters from one control plane. Maybe readers would focus more on your actual technology and not fixate on the single phrase "multi-cloud."

I couldn't help but think "I heard you like orchestrators, so we put an orchestrator on your orchestrator".

Drain you of your sanity... Face the thing that should not be.