Half of Organizations Spend More with Kubernetes, Mostly Due to Overprovisioning
infoq.com
infoq.com
For most folks here on HN, you should not care about your cloud opex cost as long as it is time benefiting. The most costly thing to waste is time.
Can you cut your cloud bill by 35% with optimal provisioning? Most likely. Can you do this cut with a greater return given all the inputs, time, wages, opportunity cost, AND what level of confidence in completion to that exact or less deadline?
The only times I've seen the numbers work out has been, growth is flattening, +$3M/yr vendor spend or $45k/month specific service spend.
Kubernetes has universally been a productivity killer in the organizations I’ve seen deploy it. It’s one of the bigger threats to the company at my current gig.
Maybe it’s just stuck in the trough of disillusionment this year, but I’d love to see a non-terrible replacement for it take over.
imho you can't really raw dog k8s unless you've got a team dedicated to building tools or, to use your analogy, packaging up sets out of the messy box of Legos and writing assembly instructions.
I'd be curious what the pain you've seen is.
Also, if you use some of the tools out there advertised to make K8s easier (operators + Helm) - you actually do have a harder time debugging the thing if you don’t know K8s well due to how many layers of abstraction those tools add.
The survey’s reported breakdown feels right – you need solid leadership to resist the urge to skimp on indirect infrastructure costs and I’d bet most of the 80% know this is a problem but have been too busy to spend time on it:
> just under a fifth (19%) had access to accurate information, while 40% said they estimated their Kubernetes costs. There was no monitoring in place for 38%.
1. The tools you want to use work with the other k8s components or things commonly available in k8s environments. Say you want a cloud-portable messaging system. You can deploy Apache Pulsar on k8s and if you use the operator, helm charts (eww) or templates it will integrate out of the box with Prometheus, cert-manager, whatever volume provisioner you are using etc. You can't get that level of cloud-agnostic deployability or integration with tools like Terraform/Pulumi because all the underlying infrastructure resources are different on a per-cloud basis. Having such a powerful generic API that properly describes like 99% of workloads is a massive game changer.
2. The skillset is transferable. If you are comparing k8s to something like ECS, Cloud Run or friends you will end up with a bunch of proprietary knowledge that is hard to transfer. Similarily your hires won't come with that knowledge either but in this day and age they probably have robust k8s skills if they are infra folk or are aware of it if above that part of the stack.
3. Flexibility. Perhaps it's greatest strength and greatest weakness is that the set of primitives chosen to be exposed by k8s (with the exception of things like Ingress that needed to get re-done) are by and large skewed towards flexibility. This means doing all manner of 'very difficult things' are now considerably easier. Conversely however this flexibility means you still need to fully describe more common patterns which simpletons mistake for being "over-complicated" when what they really mean is "less easy". Don't fall into the trap of equating "easy" to "simplicity", k8s once understood is by far one of the simplest but as a result is usually more work to get started. Good example is say Arch Linux vs MacOS, the former is simple the latter is easy.
IMO though compatibility and skill transferability (which speaks mostly to adoption rate) are separate things.
Another way of approaching it: would the cost of porting, say, an app which uses Lambda/SQS/SNS to GCP/cloud functions/PubSub be greater or lower than the cost of maintaining your own FaaS & queue/notification service on top of k8s? The answer isn’t fixed and will depend on things like your usage and staffing, and how complex your applications are, so I’d be most curious about how commonly people pick one versus the other and regret it due to some factor they didn’t anticipate.
Everything must be dockerized and dockerized stuff must run in the cloud.
Another benefit of running small, cross-functional teams. It's a lot more difficult for bloat to become overwhelming when teams have strong autonomy over their own decisions and stacks. Think this will naturally shift down as larger orgs make switch to small, 8-12 person autonomous teams.
at my last job as head of eng at a small (15-20 devs), I pushed teams to build their dashboards with business metrics in addition to the usual operational metrics. things like costs, customer frustration signals and support ticket volumes, product metrics, etc would show up there.
integrating these initially is non-trivial, but once the data is flowing, it's easy for teams to pick them up. gives a good sense of ownership, helps teams address these proactively (or build a business case for addressing them), and gives the team+stakeholders a common dataset so everyone has the same complete picture to work from.
I’ve found the opposite to be true unless autonomy is paired with accountability. Ultimately the people making decisions need to pay the costs for their choices and I’ve seen that be especially bad where someone had the autonomy to spin up a ton of resources which someone else had to pay for. This can be especially bad if that was resume-driven development where they were playing cloud service bingo trying to list experience with things they really couldn’t have justified based on the application’s actual usage.
Most places over-provision their virtual machines also, if not more so due to the less dynamic state of the lower layers.