Hacker News Reader

Comment by spwa4 | Hacker News Reader

So switch to openstack or kubernetes (with kubevirt if you want VMs). Open source. Way more beautiful design.

With Kubernetes, actually fast storage if you need it. Can scale up to AI demands if you need it.

Or proxmox or the like if you're small enough.

My beef with K8s (and to be clear, it’s the leanest cut of beef from the deli - so not much substance to it) is that unless you pay someone else to manage the Control Plane for you, you’re not only going to need to upskill your workers on K8s itself but also administering the components of the Control Plane, like HA, etcd, storage, network plane, etc.

Compared to standing up literally any Linux distro and KVM, K8s remains an overly complex PITA to get off the ground and integrated into an org on the cheap/free. In that area, it handily loses to even Microsoft Hyper-V in the “just get us going” category of business adoption/velocity.

I’d really, really like to see K8s more streamlined for initial deployment than it is. It’s getting better, but I generally still have to grudgingly recommend a premium, managed control plane for any serious deployment.

For small on-prem shops that don't really want to learn about running k8s, and have under a few dozen nodes, there are definitely slim options, like Talos is basically boot to k8s, and for single app level, there's things like multi-node k0s. Tech like this means you can reduce the control plane labor and focus on the workload.

https://www.talos.dev

https://docs.k0sproject.io/v0.11.0/k0s-multi-node

Talos is on my shortlist but its core “grease” features remain locked behind a (reasonable, but still existent) subscription, which throws it into “premium management layer” territory for me and my odd slide deck for executives. The narrative for the past fifteen years has consistently been “we have no money for what we need because we spent it all on what Gartner suggested and a consultant told us we should have”, which means we’re constantly having to not only do more with less, but also rely heavily on “pre-greased” products like hypervisors.

I don’t like it, but that’s how the current technology environment is unfortunately setup.

God help the enterprise software segment if customers realize 90% of their needs are served perfectly well with KVM+QEMU and VMs.

Word for word my experience with operating k8s.

To be clear: I like K8s! It’s fun to be able to write some YAML, apply it, and be done!

But the sheer work of getting to that point, safely and securely? It ruins the experience for me, personally.

What would you like setup to look like?

Would there be real interest in a kubernetes distro that takes IPs and a (set of) domain names, and boots up on N nodes, installing letsencrypt, so that you can do a deployment and have ingress actually working?

For homelabs and SMBs, that is definitely an opportunity. Other areas I’d like to see more growth in:

* “No-Code Kubernetes”, that lets staff design basic (or even not-so-basic) deployments using a web-based GUI. We’re seeing more of this from a “understand how things work while they run” perspective, but I haven’t personally come across any “here’s your building blocks and explainers of the environment variables, go build” solutions.

* An “ESXi-ified” K8s. Talos comes so, so close to this, but I’d love something that was as easy to deploy into production as ESXi was on SD cards. Deploy as an appliance on bare metal or as a VM, and voila, Control Plane with an IP schema, network layer, AD CA/ACME support (including Let’s Encrypt), and a basic load balancer/ingress. Changing the setup is as simple as adding a basic text file with the control plane IP and join string (worker nodes), with a simple flag to add it as another control plane node for HA.

* Renewed focus on etcd management ease. A lot of the cert track focuses extensively on etcd management through kubectl, which is an unnecessary abstraction layer for things like backups, failovers, and redundancies in smaller IT departments.

* Automated migrations. The K8s evangelists hype it as being able to manage VMs, which would be great if kube-virt was standard (it’s an add-on). I’d like to see K8s either formally integrate it into the baseline or more distros make it a checkbox option at cluster creation. Part of that should also be support for automatic deployment creations for existing VMs in a hypervisor, by analyzing current settings and suggesting the YAML or JSON to replicate that VM in K8s with appropriate IP address, current storage, and ACLs

From a tech standpoint, the foundations for K8s success have long been polished into a mirror shine. At this point it’s usability and accessibility that remain broadly unaddressed, especially if we want more people and companies using iterative, composable infrastructure.

Check out Harvester: https://harvesterhci.io/

Perhaps I'm ignorant but why would you NOT have someone else manage the control plane? It's relatively easy to switch providers and there tons of options. It's also pretty cheap.

I havn't looked about but I'd be suprised if there wasn't also a large number of companies providing on prem control plane support.

It feels like that's the direction most people are going in, but that doesn't change the fact that no one is going to trust Broadcom again after this.

After this? Many people didn't trust Broadcom even before they bought vmware. This isn't something new on Boardcoms part, though it is high visibility and so people not even aware of Broadcom before are now.

Pretty sure in Tesco's case switching to openstack is a decade long project.

I think they can do it in 5 years with some investment. Which is how long they need Broadcom to honor the current contract. The effort is mostly technical, and much of it you can just hire contractors to help.

I would bite the bullet and begin any switch from Broadcom now on the principle you can never trust the vendor again and you will just be kicking the can down the road.

Reply on news.ycombinator.com