Google admits Kubernetes container tech is too complex
theregister.com
theregister.com
It's taken us a little over a year. Partly because K8s has a steep learning curve, but also because safely transitioning services without disrupting product teams adds a lot of overhead.
The investment is already yielding great returns. Developers are happy. Actual quote: "Kubernetes is the biggest quality-of-life improvement I've experienced in my career."
The same is true for me as a DevOps engineer. I never have to write a script to orchestrate a rolling deployment ever again. I'll never have to touch Puppet or wrangle OS upgrades. I no longer need to use Terraform to scale out a service.
Now that we've completed this transition, one of my goals for the next year is to get the team to a place where we spend 75% of our time on higher-level projects -- working on developer tools and Kubernetes extensions and setting site reliability standards, instead of the firefighting and operational work that has traditionally taken up so much of our time. Couldn't be happier about this change.
... That is until management decides we're overhead and fires us to free up budget for feature developers. ;)
I'm on the DevOps side as well going through the same transition, k8s also allows insane customization, and I have some colleagues that are delaying our rollout unintentionally so they can play around with developing more tooling for deployments which is really frustrating. The k8s scene seems to be filled with constant scope creep and refactoring to get it just perfect before use. Either way, I agree the benefits far outweigh this annoyance that I've experienced. I'm so excited to work on developing tooling instead with my time.
However, I don't think we're free entirely from managing servers the old way with Chef / Puppet / Ansible, unless you're purely hosted there's still the rule of thumb you shouldn't run services that hold state in k8s. But with persistent vol's I do see that changing, though I'm not sure if everyone agree's that's a good idea.
My impression is that the primary purpose of Kubernetes is to give SRE teams political air cover to rewrite a lot of their existing processes. Whether Kubernetes is actually required for that, or even net superior seems questionable. This unsexy work becomes justifiable because it's coupled to a mainstream accepted tech modernization.
You see this same phenomenon with database migrations. Where what the team really needed to do is just rewrite an app to use the existing database properly. But no one is going to approve that work. So what happens is people convince themselves that the existing tech sucks and use that to rationalize doing the rewrite. The result ends up not always being net superior, because sure you did the rewrite but you are also eating the operational cost of integrating a new technology into the org.
We are running a handful of stateful services in K8s (things like MongoDB for which GCP doesn't have a compelling and affordable managed offering). It's definitely more complex than transitioning a stateless service, but so far our experiences with StatefulSets and PersistentVolumes have been good. And this allows us to sunset Puppet/OS management completely. I should note that we _are_ being extremely careful about backups. We also run each stateful service in a dedicated node pool for isolation. Who knows, maybe a year from now we'll be shaking our heads and saying "that was a TERRIBLE idea" but for now, so far so good.
We're running on GKE, so lots of things that would be hard in on-prem environments (ingress, networking, storage) are easy.
They hire other people with the supposedly same skillstack and then have them rebuild it from scratch.
Can you elaborate? What exactly about Kubernetes improved the developers life?
We didn't have these before. Yes, you can implement them without K8s (I have, at other companies) but to get the full set of features K8s provides, such as deploying N services in parallel, taking no more than X% of your capacity offline at a time, short-circuiting in the event the app is dead on arrival, connection draining with timeouts, you end up with a VERY complicated multi-threaded codebase.
2. Seamless horizontal scale-out.
Want to scale up your app in a test environment from 3 replicas to 6 to do some performance testing? This used to be a DevOps ticket that would take a few days -- DevOps engineer tries running Terraform, but oh no, a CentOS package update seems to have broken our Puppet manifests and we have to fix that. Now the developer makes a PR to the GitOps repo where they adjust a single YAML setting.
3. GitOps/ArgoCD.
ArgoCD is possibly my favorite piece of software of all time. It provides incredible visualizations for what's happening with your Kubernetes infrastructure. It really increases your confidence and trust in the system to be able watch a deployment rollout or scaling operation happen in real time. ArgoCD makes it spectacularly obvious when something has gone wrong -- you still sometimes need to go spelunking through the GCP console or use kubectl to inspect resources, but to a much lesser degree. I cannot emphasize enough how magical it is.
These are all things that our initial implementation has delivered, we are also planning to try to leverage K8s for things like on-demand pull request preview environments, hosted developer environments, and canary deploys that are MUCH harder to implement in a world without K8s (trust me, I've done it).
It reminds me of docker-compose file, except I can publish it too.
Just wait until the developers discover Google App Engine, Heroku, or DigitalOcean App Platform.
I think Heroku and DigitalOcean App Platform are still going to be popular for small setups (as will things like Amplify) but when you outgrow those (or realize you are paying too much for them) then Kubernetes is a reasonable option.
I do agree that kubernetes is a pleasant experience from an application developer perspective with an existing cluster, but in my experience it was not without excessive pain and long hours by those administering the cluster. A year doesn't surprise me in your case, which brings to mind this question.
Are you still using Terraform to specify your deployments? From experience, you need something there to manage all your yamls: deployments, configmaps, secrets. Especially if you have multiple environments.
Then you can focus on containers which can be run, tested and built wherever without the fear of broken updates or one thing stepping on another. We found back in the days of ansible and chef that we had very low confidence in upgrading hosts live. So we would then do immutable hosts and blue green deploy them to production. But why think in the scope of hosts and VMs when really you have some application that needs to run somewhere.
K8s IMO isn't the end all, I think eventually we will get to something that doesn't need containers at all and you run just processes. But it is a good step for now. Also once you have your stuff containerized it makes other non k8s stuff easy like AWS Lambda
Edit: Also yes you can use those to set up generic k8s nodes but when we ran bare metal we used kubeadm to make coreOS immutable nodes. I don't think that is used anymore haven't checked but really the best way to set up k8s is to deploy really thin hosts that have nothing but Docker and k8s. VMware and others have solutions for this too where you don't have to mess with building hosts.
So if you had a fleet of ten containers running the same application in a load balanced config, I'm guessing you'd need to upgrade all of them at once (with downtime) rather than upgrading them one by one (because then the database would be inconsistent)? I'm assuming that since the containers are immutable the data is stored elsewhere.
Who is preparing Dockerfiles? Developers and system administrators / security people do not generally prioritize same things. We do not use k8s for now (therefore I know very little about it), so this might not be relevant but how do you prevent shipping insecure containers?
Ansible or Puppet still excel at that kind of work.
For me the scariest part would be that it's not tech I would use (i.e. update the descriptors) regularly, so what if something goes wrong, how quick would I be able to identify the problem. I have no answer to that because I'm off the project.
It's a real risk, at least perceptually. Mitigate it by documenting the number of developers' hours you save with automation and better infrastructure. It's pretty hard to argue against a trend of increased developer productivity. Include your own hours; you're targeting saving 30 hours a week of your own time and that frees you up to improve other things even faster.
If you're determined to be skeptical of the value-add from Kubernetes, my random post on HN ain't gonna convince you :) The developer experience improvement is obvious to people actually interacting with K8s.
I do want to clarify that this isn't the only thing we accomplished in the past year. Just the thing we shipped that was our biggest priority and had (by far) the biggest impact.
Snake eating it's own tail.
But not only do I not understand K8s, I'm also an idiot.
It doesn’t end there... after complaining for a really long time two things happen. First off so much time has passed that you get these domain experts (Devops people) whose entire job is to mess with kubernetes. Second the complainers have been complaining so long that people get tired of it.
You now get people who are so tired of listening to people complain whose entire job entire job revolves around kubernetes that these people now start complaining about the complainers.
It happened with JavaScript. Javascript was around for so long people started complaining about the complainers. In fact it’s been around so long that a whole generation of people who’ve never used any other language was literally born. These people started off as the complainers against the complainers but now they outnumber the complainers so you rarely see people talk shit about JavaScript anymore.
Actually JavaScript has been around so long that the entire language has changed and part of the terribleness was fixed by making another language (typescript) compile into JavaScript.
Which brings me back to kubernetes. Kubernetes is a bad tool with no alternative precisely because it requires a 3 man dedicated team a year to get things up and running.
A good tool would be something like allows me to to get it up and running in a week just by reading some docs. Even better an hour. Could such a tool exist and replace Kubernetes? Yes. Does such a tool exist? No.
I am a complainer and you are a complainer of complainers. What will likely happen some time from now is two possible things. Kube will be so integrated into the infrastructure ecosystem that wrappers will be written on top of kube just like how react and typescript have replaced JavaScript. If that doesn’t happen then a whole new tool will replace it.
I’m sorry to say but the ideal we are shooting for here is a tool that will ultimately make Devops a general thing that all developers can deal with rather then an entire specialist team. Again no such tool exists yet but it certainly can exist, especially when the inventor of the tool has become a complainer.
If the inventor of the tool becomes a complainer, that validates the complainers. And now the complainers of the complainers have nothing left to say.
Your 5-person startup does not need a DevOps engineer. Your 50-person org should start thinking about it. Your 500-person company probably needs multiple DevOps engineers -- having every dev team independently figure out how to handle things like deployments, reliability, and security is chaotic and wasteful. There are a lot of details that only start to matter at scale, and both Kubernetes and dedicated infrastructure teams are for this use case.
Instead of complaining, why don't you build this tool? That's the problem I have with complainers.
- I can store anything in a secret? Let's have thousands of cat images. Etcd then stops working because we have over 2GB of funny cats in the key store.
- I can run a root Pod? Lets mount the docker socket and start building images with it. Oh and by the way, I never clean those up and my Node simply fills up. Also I add some additional docker networks that break Pod to Pod networks.
- Istio is nice - why we don't add automatic injection for Pods in all namespaces? Including kube-system? And then they brick kube-proxy and the cluster stops working.
- I can use validating webhooks for better security? Lets watch on all resources. To keep it more secure lets set the failure policy of the webhook to Fail, so we never admit any modification without the apiserver to make a call to out webhook. Whats that? My single replica webhook has was evicted from the Pod (we didn't add any resource requests and limits) and now it cannot even be created or scheduled because kube-controller-manager and kube-scheduler cannot update their lease and they lost leadership and now are idling, effectively bricking the entire cluster.
Google would reduce the pain points with this change, however they would still face countless other issues with Kubernetes.
However playing the devil's advocate here: If you actually took the steps of learning the basic abstractions, then for me it's really hard to see what you could still get rid of.
If you actually go all-in and fit your application to the principles of Kubernetes-native applications (instead of the other way around), then it works nothing short to amazing.
We're running 120 microservices in GKE and the difference to our custom-built setup before is night and day. I let my Infra team go surfing together for two weeks because without changes it flies mostly on autopilot.
Let's not kid ourselves, distributed computing is _hard_ and Kubernetes is a testament to that. I'm not saying it can't be made more accessible by further standardization, but there are fundamental limits to how easy it can be made.
Which by the way is leading to my only pet peeve with it: I feel most of the complexity of K8S comes from the fact that it got hyped as an enterprise product and then lots of features were built that support shoving your non cloud-native workload into Kubernetes even if it was never designed for it.
If you don't do or need all of that, the amount of interface, complexity and footguns shrinks significantly. Maybe it's time to better pull them apart in the documentation.
This argument basically sums up to "Developers just need discipline, and stop blaming the tools". While this is a sound argument on paper, the intrinsic complexity of software systems make it hard to pin the blame on developers. BTW This is the same argument Uncle Bob makes which is not so popular with many mainstream developers.
You're right about feature creep in k8s though.
If your goal is to build highly reliable and available services to end users that are secure and scalable with a team of more than 10 engineers, eventually you will run into more than 50% of the concepts in Kubernetes anyway and end up re-inventing them.
Scaling up and down, node draining, finding out whether services are healthy, RBAC, resource distribution, secrets management, service hardening, introspection capabilities, explicit declaration of dependencies and endpoints and many, many more.
My point is: Sure, if your goal isn't that, it doesn't make sense to start out using Kubernetes.
But if at least eventually that's what you need, imho it's way preferable to just learn and apply well proven abstractions instead of reinventing the wheel along the way and end up with a less maintainable, capable and standardized solution you won't find anyone for maintaining.
If I hear about some of the comments here suggesting to "just spinning up docker-compose with Traefik in front" (disclaimer: I really like Traefik), then that reminds me of how some of the ops mess started that I historically had to care for.
It’s always easier to shift the blame somewhere else.
Thats why some radical but correct concepts are so hard to push.
For example you still need a way to get BEAM onto hosts, still need to manage the OS on the host, still need to setup networking, RBAC etc.
In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident:
GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind
One of the goal of K8S is normalization/standardization of a complex topic to better share knowledge
Why would someone want to store non-secret information as a secret?
— Douglas Adams
If you have a managed object store or even relational database outside of k8s, the thought of storing arbitrary data in secrets probably doesn't come to mind. But if your enterprise spools up a cluster and tells you to use nfs PVCs with no other storage solution, suddenly you might start getting creative.
Top reason given to me by developers: "I don't want to spend time thinking about the distinction."
Just in case you haven't figured out the proper way to do this, you should use docker:dind.
Sure just put your secret data on a file then we'll use your file name as the key of the secret.
Cronjobs sometimes have weird bugs as well.
A lot of its complexity is due to the fact it's an evolving system, that's fine. But I see that some things end up way more complex or unreliable than it needs due to overengineering or use cases no one needs
For our current stack, the answer has been to make the entire business application run as a single process. We also use a single (mono) repository because it is a natural fit with the grain of the software.
As far as I am aware, there is no reason a single process cannot exploit the full resources of any computer. Modern x86 servers are ridiculously fast, as long as you can get at them directly. AspNetCore + SQLite (properly tuned) running on a 64 core Epyc serving web clients using Kestrel will probably be sufficient for 99.9% of business applications today. You can handle millions of simultaneous clients without blinking. Who even has that many total customers right now?
Horizonal scalability is simply a band-aid for poor engineering in most (not all) applications. The poor engineering, in my experience, is typically caused by underestimating how fast a single x86 thread is and exploring the concurrent & distributed computing rabbit hole from there. It is a rabbit hole that should go unexplored, if ever possible.
Here's a quick trick if none of the above sticks: If one of your consultants or developers tells you they can make your application faster by adding a bunch of additional computers, you are almost certainly getting taken for a ride.
It's a somewhat odd solution for a too common problem, but any solution is still better than dealing with such an annoying problem. (source: made docker the de facto cross-teams communication standard in my company. "I 'll just give you a docker container, no need to fight trying to get the correct version of nvidia-smi to work on your machine" type of thing)
It probably depends on the space and types of software you 're working on. If it's frontend applications for example then its overkill. But if somebody wants you to let's say install multiple elasticsearch versions + some global binaries for some reason + a bunch of different gpu drivers on your machine (you get the idea), then docker is a big net positive. Both for getting something to compile without drama and for not polluting your host OS (or VM) with conflicting software packages.
The list goes on and on, it’s bizarre to me to think of this as the true, good way of doing software and think of docker as lazy. Docker certainly has its own problems too, but does a decent job at encapsulating the decades of craziness we’ve come to heavily rely on. And it lets you test these things alongside your own software when updating versions and be sure you run the same thing in production.
If docker isn’t your preferred solution to these problems that’s fine, but I don’t get why it’s so popular on HN to pretend that docker is literally useless and nobody in their right mind would ever use it except to pad their resume with buzzwords.
./configure
make
At a prompt, then install missing libs. Unless you have to maintain updates regularly, “It’s just so hard” seems like a damn meme.
Refactoring your application so that it can be cloned and built and ran within 2-3 keypresses is something that should be strongly considered. For us, these are the steps required to stand up an entirely new stack from source:
0. Create new Windows Server VM, and install git + .NET Core SDK.
1. Clone our repository's main branch.
2. Run dotnet build to produce a Self-Contained Deployment
3. Run the application with --console argument or install as a service.
This is literally all that is required. The application will create & migrate its internal SQLite databases automatically. There is no other software or 3rd party services which must be set up as a prerequisite. Development experience is the same, you just attach debugger via VS rather than start console or service.
We also role play putting certain types of operational intelligence into our software. We ask questions like "Can our application understand its environment regarding XYZ and respond automatically?"
I built a service that is installed in 10 lines that could be ran through a makefile, but I assume specific versions of each library of the system and don’t intend to test against the hundreds of possible system dependencies combinations or assume it will surely be compatible anyway.
The dev running the container won’t building their own debian installs with the specific version required in my doc just to run the install script from there, they just instanciate the container and run with it.
At the risk of nitpicking, docker images aren't the equivalent of VM images, as they don't include a kernel.
Docker is not virtualization, it's just an abstraction that makes some Linux process isolation features easier to manage.
It also allows you to bundle whatever dependencies you have in the same bundle, but that is not the same as having a VM.
I used to do UNIX integration work in the late 1990's early 2000's and containers weren't really a thing. So you had to make sure libs from one program didn't crap on another program. And developers had to be conscious of what dependencies they included in their code. Nowadays they don't have to care as much because of containers. Every program can have its own dependencies. Thereby solving the integration problem.
A better solution would be to actually integrate programs and their dependencies into working systems, but no one has time for that. Software bloat is fine. Computers are cheap and fast. And actually understanding what we're doing would be too expensive. So just wrap all your have finished crapware up in a giant black box and dump it on a server.
What I'm not interested in, is this kind of walking uphill in the snow both ways:
> So you had to make sure libs from one program didn't crap on another program. And developers had to be conscious of what dependencies they included in their code.
ie. having to understand what everybody else is doing in order for my software to run properly. No thanks. That's not why I'm here.
I'll put the exact dependencies I want, in the versions which work best for my software, into a Docker image or whatever tool offers a similar level of isolation, and I'll be working on my code while everybody else spends their time fighting over the ABI compatibility of C system libraries.
Separating the dependencies between programs allows you to test and release independently to allow incremental upgrades. IMO that is better.
And don't even get me started on having instances labeled "large" that have less memory and CPU capacity than my personal backup laptop (currently on loan to my 8yo for COVID reasons)...
But you can use other cloud providers with better value.
People have since bought into the marketing reasons for 'being in the cloud' and having 'infinite scalability' but that largely misses the point (and the pain) that caused many of these technologies and patterns to be developed in the first place.
The best example of how to scale without buying into this pattern I know of is Stack Overflow. At least circa 2016 - https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
Maybe if your app handles < 10k concurrent connections. Otherwise it is the most cost efficient solution and exists because it solves the scaling problem in the best way as of today.
But...
Splitting a horizontally scalable workload across a dozen virtual servers that are barely larger than the smallest laptop you can get from Best Buy, you are just creating self-inflicted pain. Chances are the smallest box you can get from Dell can comfortably host your whole application.
The fact remains the odds of you needing to support more than 10K simultaneous connections are vanishingly small.
Mind you, that's still Docker, though, not Kubernetes.
> If one of your consultants or developers tells you they can make your application faster by adding a bunch of additional computers, you are almost certainly getting taken for a ride.
Eh. There's a redundancy play in there somewhere too, if you know how to pull it off. (Big if.)
Imagine of your only unit of compute is a single bulky machine. You don't fully saturate it, but you need a second machine to avoid downtime anyway. Now you spin up a second or third service and suddenly you need 5 or ten machines and your compute utilization is 20%. You can pack things in tighter. But then you have a knapsack problem, and that's easier to solve efficiently with many small blocks even if it costs you a 1% overhead or whatever.
I've seen this (a long time ago) in the education world market. Very small school with a STEM program. They had specific scientific software they wanted undergrads to use (and some of it was pretty proprietary and used to interface with lab equipment) + a pre-configured IDE.
Instead of going through the compatibility matrix of OS and their versions they just gave all students a VM image that would "just work". Everyone could bring in their own devices and as long as you could run an hypervisor everything would "just work".
That said, I haven't really used docker as a runtime container.
Cassandra is a big PITA, but it hasn't gone down (knock on wood) in the 6 years I've been using it. PARTS of it have...
How many distinct services do you have in your monorepo? A monorepo is fine...
until it isn't.
There's your first miss.
How big should the one server that serves the whole of netflix be..?
You can pretend to be Netflix or Google, and build your tech-stack like they do. Or you can stop wasting your resources setting up a tech stack that you’re never going to get a return of investment on.
Apparently netflix has that many customers. Then again, if you split Netflix into regions and separate all the account logic from the streaming, the recommendations engine and the movie-content, you could perhaps run the account logic for one region in one server.
I'm not interested in being "not lazy." I only care about user value and ability to provide user value (tech debt/cost/velocity).
For the price of an M5A.16xlarge to get those cores, I can get like 27 m4.larges and have way more fault tolerance.
I think your disdain is misplaced.
But it's not the case in linux, or at least non-enterprise / ldap linux. Installing mysql/redis/elastic/dotnet core/etc on each machine may be different, and has different installer available.
With docker I just need to instruct them to install docker, setup docker compose and everything is handled via containerization.
Which we later replicated in Red-Hat with RPM.
Bare bones OS install + bunch of OS packages => done.
And in what concerns containers I was working with HP-UX Vault in 1999.
Don't you just end up with a less hackjob version of a container when you do that?
If you're looking to have your small app eventually grow into a large one, read up on K8s and just make sure you're not blocking future-you from making your app work on it. E.g., work well in a container (which is useful for automated testing, deps management, etc), have a simple 'ping' endpoint to make sure the app is up, have a better config story than "recompile to change these variables", use a logging library, and tolerate any other services you're using to sometimes be down.
All useful things for a grown-up app to do anyways, all a bit of a PITA, and all better than trying to operate an app that doesn't do them.
If you have one monolithic backend service (and most web applications really should start out this way), Kubernetes offers almost no benefits over alternatives.
I don’t think it suits all teams and use cases, but for us it’s absolutely fantastic and without going down the rabbit-hole of cloud-provider specific tools and recreating half the issues it solves, I’m not super sure what we’d use.
I've already climbed most of the learning curve so YMMV, but as a team of one and dozens of WordPress, MySQL, and bespoke app servers, kuberenetes makes ops manageable so I can spend time on things that really matter.
Deploying new web apps is trivial, declarative manifests are easy to reason about, TLS certs are issued and renewed automatically (cert-manager), backups are cheap and reliable (daily GCP snapshots), making changes to the cluster via declarative terraform is a breeze, etc etc. No way I could manage all the ops without leaning so heavily on the core foundation provided by k8s.
That issue is not endemic to Kubernetes, but rather to any larger system past a certain age, you learn stuff as you go along and would do stuff differently if you did it again today - but you can't easily, because you cannot break compatibility for everybody using your stuff.
As a concrete example from the Kubernetes world, there is a talk by Tim Hockin [1] about how today, they would fundamentally design the api-server differently and base pretty much everything on CRDs.
After a week of almost full-time work, I threw in the towel. Admittedly, I also had to learn concepts like reverse proxies alongside, too, so I was by no means well-equipped to begin with.
Yet, tossing together some docker-compose.yml files and "managing" them with a Python script has worked very well. Kubernetes really scarred me in that sense, and I am healed! Also, Caddy has helped me in actually enjoying configuring the webserver.
Not every addon/tool for the k8s ecosystem is worth it. I also don't bother with the ever-growing list of service meshes... not enough value to me for the overhead.
K8s is definitely the simpler alternative for me but there is still a lot of essential complexity in k8s due to the nature of the problems it's trying to solve. Mostly I like building on top of a solid foundation of standardized k8s API objects (pods, services, volumes, etc).
Tldr; Bring in only the add-ons and tools you really need so you don't add more complexity than necessary. Don't get swept up in the hype and marketing from other devs and cloud vendors.
Also, I don't much like Go's templating syntax.
You are absolutely spot on because this is how not to pass the behavioral interview for Engineering Manager.
There are very strong financial incentives for every individual developer and sysadmin to adopt Kubernetes, regardless of the impact it has on the organisation as a whole. In a sense this is engineering reaching the level of corporate maturity of the sales department who will optimise everything for their commission regardless of the organisations ability to deliver it at a profit, or even at all.
Then that organization is doing a terrible job of aligning incentives. I'm guessing their pay structure isn't terribly merit-based nor high enough that people aren't constantly thinking about other jobs.
If this is about FAANG (your comment wasn't, but others were), perhaps part of this is exposing larger problems in many smaller orgs. (note: I'm ex-FAANG and happily so)
Nomad is way simpler to get a cluster up and running, has a great configuration syntax (I'll take HCL over YAML anyday) and had first class Terraform/Consul/Vault integrations.
Onboarding devs is fairly straightforward, if they can write a docker-compose.yml, it's an easy transition to a nomad job specification.
It took me by myself ~4 months to get our current hashistack(Vault/Consul/Nomad) stood up using Terraform+ansible. Two members of my team have been working to replace the hashistack with a self hosted K8's deployment and they just went over the 1 year mark and we still do not have something capable of hosting the workload currently running on the Hashistack.
This got a little long winded but I feel like this "it's docker compose or K8's, take your pick" mentality had led to a bunch of needless time being spent by smaller teams/companies on solutions that just aren't right for them.
Also you can get red/green deployments and rolling deployments with little to no effort, which can be very nice, nice to have.
This feature helps a lot with that problem by bringing GCP closer to where AWS has been with Fargate. k8s will still be more work than using AWS ECS but it might also be preferable if you dislike using the provider’s components and want the control of, for example, doing your own load balancing and storage management.
I like Cloud Run for the same reason because I can use it without needing a lot of devops skills in my team or without sacrificing my own time (because I have those skills but have more valuable things to do). It allows me to focus on keeping my CI/CD pipeline (cloud run sets that up with a button click) busy with new functionality. And our hosting cost are close to 0$ because we stay below the freemium layer until we actually need to scale.
Edit. corrected the typo 700->70
There's (almost) no limit to what you can run in one cluster too, and Kubernetes namespaces can help to separate different environments to allow for sharing.
Cloud Run sounds like the perfect solution for your workloads though!
for these case, i still have gke cluster around.
I previously avoided Docker Swarm for ages since I assumed it involved the same level of complexity as k8s. I also initially figured that managed k8s would be a safer bet than managing my own Swarm cluster, but if you’ve used anybody’s managed k8s (or read https://k8s.af), you’ll realize that every cloud provider has their own closed source fork of k8s with plenty of nasty bugs that you can’t do anything about.
Hashicorp's Nomad is the best choice on the complexity for features scale IMHO, and that's why i'm writing an article how great it is, how easier some things are and what's missing compared to Kubernetes.
My only complaint with Swarm is there isn't an easy way to expose containers directly on the network (like host networking). I have a few containers (wireguard and minidlna) which need this, so those are running through docker-compose. I've tried macvlan but wasn't able to get that working in Swarm mode.
For long I was scared of it because so many people say it's crazy complex. But actually it took no more than a dozen of hours for me to learn it and get a working setup on aws.
Maybe being full stack and having a strong knowledge of Linux and Docker helped.
Now I'm not pretending to be an expert with it, and there are certainly traps and mistakes that I didn't experience yet. But I don't understand what people find to be hard about it.
Also why Docker in the first place? I’m genuinely wondering - in the stacks I run (Express / Python) it doesn’t seem necessary at low scale. Elastic Beanstalk, Heroku, Digital Ocean etc all offer facilities for single-command deploys that work out of the box.
Docker make it easy to run the same version of code in different places and let’s things run next to each other without version conflicts.
Also, I think you’re in a very small minority not to care about $720/yr increases in your hobbies.
This way:
- I can easily move from dev to prod by using the private container registry
- I have apt-get on the server and just use the default Ubuntu
- No distributed/network state
I used to use Core OS, but all these container OSes are here today and gone tomorrow, and normally have their own config standard. At least with Ubuntu they have LTS and a bunch of Google pages for fixing stuff.
In places with 10+ developers and 10+ services on a single cluster it works surprisingly well from the user (developer) side.
Definitely a learning curve but honestly not bad if you like ops. Absolutely possible and beneficial for any size team. As always, it depends on what your goals are and how you want to use k8s.
If you were self-hosting k8s, then I'd agree with you.
So, not simple.
Google did not say "Kubernetes is too complex" but rather, they are making this new tool - called Autopilot - that is an abstraction layer on top of Kubernetes for certain types of applications / companies.
This Autopilot system still uses Kubernetes AFAICT.
K8s is too complex, and I say it as k8s dev(ops) who dreams the next great thing will come soon which will save me from piles of YAMLs and will bring back programming fun ;) /s
it's just that a lot of people think they have to use k8s because it's trendy even it doesn't fit at all their needs and scale. Then they whine it's too complex.
classic PEBCAK issue.
But it is a surprisingly negative headline. Autopilot sounds like a cool tool to simplify container orchestration!
That's the Register's schtick, they're snarky about everything.
The problem for me is that these project are not under full time development. Most of them see maybe one or two deployments per year. It seems like every time I do a deployment, some part of the YAML config structure has been deprecated, with no clear path on how to migrate. In the end I have to spend a day figuring out how to rewrite the YAML configs to do _exactly the same_ as before. This is really hard to explain to a customer.
But it's not just K8S itself that suffers from this, third party components do this too. Stuff like ingress control is a nightmare, I've seen the nginx ingress controller (not to be confused with the 'other' nginx ingress controller) grow from a 50 LOC base config to multiple KLOC of YAML config you are supposed to pipe to your production environment. The ACME/LE extension (cert-manager) is even worse, the default config is 26KLOC long! [0]. And you are supposed to pipe this config straight into your environment. The amount of new and amazingly complex components that are added with each new release is staggering.
If you are an average Joe developer and you want to use containers in production, just stick with a machine with docker-compose on it. It's a lot easier to maintain and has far fewer surprises down the road. It is also much easier to make a quote on maintaining a VPS box that having to get a crystal ball to predict when Google will deprecate parts of your setup.
[0] https://github.com/jetstack/cert-manager/releases/download/v...
This is exactly what I am doing. It's easy to configure and almost same configuration between local test and production machines. The only downside is it's hard to monitor whether service is up without additional tools, which I lazy to research.
Additionally if you don't have a good enough ci cd yet you can deploy with volume-linked container.
Simple load balancers without auto-scaling work fine!
For smaller non-critical services sometimes even one reliable server is an acceptable trade-off.
Do most companies really need kubernetes given the complexity and difficulty it introduces?
We have chosen k8s and I would again, because its nice to use. Its not necessarily easier, as you point out, the complexity of managing the cluster is considerable. But if you use a managed cluster like EKS or DO's k8s offering, you don't have to worry too much about the nodes and the unit of worry is the k8s config and then for deployment you can use Docker.
I like Docker, because its nice. Its nice to have the same setup locally as you have remotely.
In my experience the tooling around k8s is nice to manage declaratively, I never liked working with machines directly because even tools like Chef or Ansible feel very flimsy.
The other thing you can do is run on ECS or similar, but there the flexibility is a lot lower. So k8s for me offers the sweet spot of being able to do a lot quickly with a nice declarative interface.
I'd be interested to hear your take on how to best run a small cluster though.
For smaller setups (say 1-10 services) I'm quite happy with cloud config and one VM per process behind one load balancer per service. It's simple to set up, scale and reproduce. This setup doesn't autoscale, but I've never really felt the need. We use Go and deploy one static binary per service at work with minimal dependencies so docker has never been very interesting. We could redeploy almost all the services we run within minutes if required with no data loss, so that bit feels similar to K8s I imagine.
For even smaller companies (many services at many companies) a single reliable server per service is often fine - it depends of course on things like uptime requirements for that service but not everything is of critical importance and sometimes uptime can be higher with a single untouched service.
I think what I'd worry about with a k8s config which affects live deployments is that I could make a tweak which seemed reasonable in isolation but broke things in inscrutable ways - many outages at big companies seem to be related to config changes nowadays.
With a simpler setup there is less chance of bringing everything down with a config change, because things are relatively static after deploy.
But I grant you, it has been a very complex, painful journey to get this working right, and in fact I'm still making tweaks and adjustments. Plus, often times it's really unclear if I should scale vertically or horizontally.
I guess the ultimate goal of services like GKE autopilot is for you not to have to worry about kubernetes at all, just give them your workload and have it run on whatever resources they think is appropriate?
I do think it's important to recognise though that there are lots of ways to host services, and simpler with less abstractions is often better and more reliable and certainly easier to debug when things go wrong.
Docker introduced a great level of abstraction and reproducibility over platforms. However, Docker (or OS-based containers) are the most atomic unit of computation on Kubernetes. Which causes centering scaling on the Instance, instead of the Application or even the functions.
This leads to a lot of unintended side-effects. Of which complexity is the most evident since now you need to handle scaling, monitoring and compliance over the VM layer rather than on the functions or the app.
I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit to allow proper scalability and a simpler and more powerful system on the long term.
Docker's co-founder agrees:
"If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing."
The scaling of their primary business (amazon.com, google.com) is at least partially an infrastructure problem - so they had to solve this anyways. Why not try to sell it too?
But it's very rare that you can scale a system by only scaling infrastructure. Ironically, probably the only way this will work is if you scale vertically - which means you don't need K8 and want to avoid the cloud like the plague.
There are all types of application-specific consistency issues that need to be treated as first class parts of the system for horizontal scaling to work.
That should be why Red Hat took an early investment in Kubernetes. Operating systems did matter for them. Applications did not.
> I believe a VM centered on the app (WebAssembly VMs) or functions (serverless approach) is the right computation unit
I agree with that. The thing I worry about is a k8s variant like Krustlet, running Wasm instead of container, might introduce the same complexity into the Wasm world. We need a more app-centric scaling solution.
Are we just making rabbit holes out of rabbit holes of abstraction using kubernetes?
I just use and push code to Heroku and I'm done for the day, simple. NoOps I call it.
I wish more tools and platforms were like this.
Isn't this just labeling the knowledge that you don't have (and could probably read up on) as potentially unnecessary complexity?
I mean, every time I hear about embedded, I keep hearing about byte boundaries, RTOSes, compiler chains, musl, JTAG, and a million other things that make my mind numb. But I assume embedded engineers need to know all those things, because of the constraints unique to their field. Some of them could be just "a complex way for engineers to keep themselves spinning their wheels and not actually working on an application", but a lot of them provide value in the real world and solve some specific problem.
Cloud infrastructure management frameworks are the same.
Because everyone else is doing it.
Things have gotten so complex that we need complex solutions, but w're afraid to make and own the solution ourselves because of cost, focus on core business or recruitment/knowledge concerns. So we need to find the next best open source solution to leverage the collaborative effort to reduce time and cost and have a large community to fall back on for support. And because everyone jumps on board of the same train all use cases must/will be accounted for (else the solution will fade into a niche), meaning the solutions becomes a problem in and of itself.
Repeating the cycle once again.
Running a cluster across your IOT fleet (1k+ devices; 5-10 apps each) gives a nice interface for pushing out tasks, choosing applications for a device, configuring supervisor-device relationships, etc. It turns out from scratch Docker containers on ARM are very portable.
I think you’re being dramatic — embedded is notorious for crazy builds of weird config flags to even get “hello world” to compile, and you’re waving your arms pretending that concepts from Erlang abstracted away from the language are too much.
Docker is just cgroups and namespaces, with a zip file of code. More or less literally.
Orchestration is the same mess it’s always been — back to at least the telephone days, when Erlang used the same concepts.
When you start having a lot of pieces to manage eg. cache, database, auth, multiple applications then Kubernetes comes into its own.
Because then you can scale, monitor, trace, debug, log, backup, audit, encrypt and visualise all of those pieces in exactly the same way.
And do it irrespective of which cloud you use or whether it's even in the cloud at all.
I’m not telling people what to do, if you like K8s or Docker or what have you knock yourself out, and I mean it. For example, people keep telling other people on HN not to use React and that’s a hill I’d die on - and could write a dissertation defending it. I’m just wondering what the dev experiences of others are so I may learn from them.
All the tools you mention are not integrated into your app neither change how you write code. In fact your application shouldn't even know what service mesh you are using.
I can write a back end application once and then have it run with istio/linkerd/traefik/whatever with zero code changes.
That doesn't happen in the Javascript world. The choice of framework directly affects your code.
https://azure.microsoft.com/en-us/services/app-service/conta...
Edit: fixed name
I just enter my Honda Civic and drive to my office. Simple. I call it "Sedan".
(Honda Civic driver looking at an 18-wheeler in the highway and not understanding why somebody would use that)
In all seriousness though, Kubernetes solves a specific set of problems. Just because you personally don't have these problems, doesn't mean that Kubernetes is bad, or that people who use it, don't need it.
Lens has been a huge boon for helping us manage our kube cluster and regular devops operations, and even 15 minutes with it helped me grok a number of complex kubernetes concepts that I've struggled with for awhile now.
Everyone who works with k8s for a living should at least know of this tool imho its fantastic with prometheus
As for cloud tech: https://www.amazon.com/Mastering-Kubernetes-container-deploy... will cover 90% of it for you.
This is sad.. I don't understand why providers do it. It makes it very expensive to run small staging clusters. There is some reference on the current state here:
https://docs.google.com/spreadsheets/u/0/d/1yhkuBJBY2iO2Ax5F...
The pricing is... not cheap: https://cloud.google.com/kubernetes-engine/pricing
Comparable-ish with Fargate. You probably wouldn't be using this with an eye to save money unless you think (or have measured) that you'd spend more on operations, security or compliance otherwise.
Direct link to the docs: https://cloud.google.com/kubernetes-engine/docs/concepts/aut...
Google has a tool by the same name (autopilot) internally that does close to the same thing. There was a paper published on it 10 months ago talked about here: https://news.ycombinator.com/item?id=22980467
But likely they shared little in common.
Things are always fine when you're starting, but if you don't understand the system it's hard to troubleshoot when it breaks. Understanding Kubernetes is hard. Obviously there are ways to outsource cluster management, but we don't have the budget of a VC funded startup.
We ended up choosing Hashicorp's products -- Nomad for orchestration, Consul for service discovery, and Vault for secret management. Each one is just a binary with a <20 line config file, and it takes ~two days to read and understand the docs well enough that we have been able to troubleshoot issues quickly.
IMHO, most of what K8s offers can be obtained cheaper and in a simpler way using more "traditional" DevOps approaches and systems. I'm currently operating an IT infrastructure consisting of more than 20 different components, using only basic Linux technologies and open-source packages (ssh, iptables, ipsec, ferm, (r)syslog,...) plus Ansible to orchestrate it all. Never encountered a problem that I wasn't able to debug and fix within a few hours, and managed to have more than 99.99 % uptime so far. I understand that this approach might not work for large companies, but it seems to me a lot of startups are going down the K8s route just for the sake of it, and their DevOps processes become incredibly brittle and slow as a result.
1. Someone has an idea. It's alright. Really good for their use case. Someone else hears about it, likes it, and adapts it to a similar use case. So and so forth until the idea has a large user base
2. Employees of large companies hear about the idea and implement it
3. Marketing gets a hold of the idea, gives it a flashy name, and uses it in promotions
4. A majority of the loudest voices in the industry get on board so everyone starts forcing this idea on every imaginable use case
5. A lot of people realize this is all too much extra work without much benefit so they start looking for more appropriate solutions
6. return to 1
It's happened with mainframes, object oriented programming, services, micro-services, web-all-the-things, blockchain, steaming data (kafka), sql, nosql, containers, agile, VMs, cloud, and many other things.
It's just technology people are the worst at getting caught up today's fad and I wish we could try to not do that as much.
Most of the software tech world is bullshit. Most best practices are bullshit. And, if I were feeling a bit conspiratorial, I'd say the large tech companies do this on purpose. They promote fads that overburden any smaller company with more modest budgets, thus keeping competitors at an arm's length. Resume-driven-development plays a role in this as well.
A lot of companies following the FAANG cargo cult are digging their own grave and don't even know it. They don't realize they don't have the manpower for microservices. Or the calendar time to make it work and still get product out the door before their competitor that doesn't even unit test eats their entire lunch.
After a half-decade in the industry, I am beginning to realize there is a lot of bad engineering hiding behind marketing and emoji. So many of these web tools have nightmarish interfaces and add complexity that, for most of the industry, is unnecessary. But their docs pages are full of rocket ships and confetti.... I swear, seeing a page with rocket ship emoji has become such a turnoff. Why do we need our engineering handed to us sprinkled with decorations like a cupcake?
And the imposter syndrome that the entire industry seems to share dictates that a large percentage of developers feel like they need to be using these tools as a badge to wear saying, "Yes, I am with it and hireable."
Meanwhile, the technology itself keeps churning because there is now a profession full of people believing that creating and open sourcing the Next Big Thing in ____ Technology is the best way to move their career forward. People keep reinventing wheels because everyone is focused on making a name for themselves with the new; no one notices the person doing mundane maintenance on Rails or whatever.
I think it's been this way since the 70s/80s to some extent, after having done some reading on the history of the profession. I think it's just scaled with the number of programmers and the Internet has applied its intensification effects.
1. Build an opinionated solution that you control fully (e.g. difficult to fork).
2. Convince everybody to adopt it. Nobody gets fired for choosing $bigcorp
3. Grow it complex and expensive to maintain over time.
4. SELL software and services to manage its complexity.
...and the industry is getting more marketing/fad/resume-driven every day.
It makes a lot of sense to build portable infrastructure, but you can scale a long ways with much simpler technologies.
- App Engine has a bunch of weird limitations and slow deploy times. Qualitatively, it feels like the spotlight has moved on.
- Running my own compute instances felt like reinventing Kubernetes, especially once you roll your own deploy mechanism and throw load balancing in the mix. I also don't buy that it's easier.
- Cloud run is promising but for database heavy apps it's a non-starter.
- GKE was pretty smooth. It feels like it gets a lot more love than App Engine. The UI was functional with lots of depth. Once I push a docker image, GKE updates the nodes to serve the latest version. Load balancing was a matter of ~3 yaml files at 10 lines a pop.
It's an insanely complex solution to a very niche problem - scaling stateless web app backend nodes written in scripting languages.
Stray even a little bit off the garden path and you start feeling pain.
Containers add vast complexity, add another layer of complexity on top.
This was years ago, so maybe they greatly simplified things. But somehow I doubt it =/
[1] " Docker, by default, punches massive holes through your firewall in non-obvious ways. People don't realize that with a default Docker configuration, containers are ignoring any normal firewall rules you may have setup with iptables or ufw." - https://news.ycombinator.com/item?id=25834444
So far that's Docker, or containers to put it more generally.
Now if your web page is so amazing that it receives a lot of traffic, your little container is going to get overwhelmed. And if it falls over, then it's dead and nobody can see your web page until you bring the container back up. Fortunately there are tools that let you manage this aspect of the image, called orchestration. You can tell orchestration tools how to figure out if an image is unhealthy and needs replacement, and if it falls over whether to bring it back, and importantly, how many copies of the image to run to handle the traffic. And if you need to push an updated image with an updated web page in it, how to gently make that new container available to the world without interrupting the traffic.
There's more to orchestration, you also tell containers how to talk to each other if needed, how to manage secrets, encryption, load balancing. There are lots of aspects of hosting that fall into this.
The two main orchestration tools I know of are Docker Swarm and Kubernetes. Docker Swarm is bundled with Docker already. It's pretty easy to shift from normal Docker use to Docker Swarm use, it works well enough for small-medium deployments. Kubernetes is a tool for much larger and highly flexible use cases, and it has a lot of levers and buttons and swiss army knives with its own swiss army knives. Many aspects of Kubernetes like the load balancing and secrets are all pluggable and you can use different tools in there.
Now you're at this article's topic, which is Kubernetes. K8s as it's called has a larger mindshare of the ops world, therefore everyone wants to use it, but it's very complicated, so a tool has been introduced to try to simplify it.
Kubernetes basically lets you define your references not as "c:\jon\reports\fy2020_final_final_2_comments_review_Bob_final.xlsx", but as "fy-report", with "fy-report" being defined elsewhere.
This is required to help you run the same program in different circumstances without breaking everything. You can say "run it on this slow computer with this test data", or you can "run it on many big computers with real data", but the program is exactly the same.
What makes it so complicated is that Kubernetes tries to abstract everything a given program would need, so it's not just references to external files, but practically the whole computer with all its network connections that must be defined elsewhere using Kubernetes' special language.
A lot of this special language is the same for 99% of programs, like in Excel you want your VLOOKUP to work on a $-pinned range with the last parameter set to FALSE 99% of the time. This makes people make the same stupid mistakes and finding them is hard.
And of course, this special language means you have to relearn a lot you know about running programs on computers, like when you move from Excel formulas to VBA.
* Built the app (into a self container .jar, it was a JVM shop)
* Put the app into a Ubuntu Docker image. This step was arguably unnecessary, but the same way Maven is used to isolate JVM dependencies ("it works on my machine"), the purpose of the Docker image was to isolate dependencies on the OS environment.
* Put the Docker image onto an AWS .ami that only had Docker on it, and the sole purpose of which was to run the Docker image.
* Combined the AWS .ami with an appropriately sized EC2.
* Spun up the EC2s and flipped the AWS ELBs to point to the new ones, blue green style.
The beauty of this was the stupidly simple process and complete isolation of all the apps. No cluster that ran multiple diverse CPU and memory requirement apps simultaneously. No K8s complexity. Still had all the horizontal scaling benefits etc.
At work, we have had multiple big and bigger incidents inflicted by the "share everything" nature of kubernetes. Sadly, it never occurred to the devops team that something like this is possible. Same goes for the HN crowds here who just assume that a container scheduler is a must, be it kubernetes or nomad or docker swarm.
I blame this on Google Cloud brainwashing. Let's see how long it will take for Thomas Kurian to sunset https://cloud.google.com/compute/docs/containers/deploying-c...
If you're going from "our app is running directly in tomcat in our on-prem data center" to "we want to move to a mirco-services architecture with a service mesh, canary deployments, multi-cluster observability, fault injection, automatic global failover, etc...", you're almost certainly going to have a bad time.
I really like the concept of "innovation tokens" [0] here. Pick one or two big innovations at a time and don't add more until you're comfortable with what you have. Get your app running in docker first, and then get it running in a basic k8s setup first instead of leapfrogging to the more advanced stuff. Chances are if you didn't need a service mesh for your original non-k8s deployment, you probably don't need one for the v1.0 first pass of your k8s rollout.
We have not had any major k8s incidents in the past 12+ months of running our own HA cluster. We even run HA Postgres using local volumes. We've pretty much moved everything over and couldn't be happier.
We have red teamed disaster scenarios, we can bring the full cluster up on another cloud provider using vanilla k8s in about an hour (including a restore from Postgres S3 backups + wals). And this is all with a very small team.
The "declarative" part of Kubernetes control buried in YAML seems to fail to live up to that label.
Is there any literature that shows that the cost savings from packing more than one service per VM are significant enough to outweigh the cost of the messes created by middling developers who have trouble reasoning about the functioning of single process application at non-Google companies create when they think they should do what Google does?
Imho it was very simple to understand all the components. But I can't deny that it does require a solid understanding of sysadmin concepts and containers to get ahead.
I was quite proud to have grasped kubernetes within a few months, without any formal training. No finished education in my country and not from an English speaking country. So I often struggle with technical docs that are using too much academic english.
Still today when new issues arise I can relatively quickly understand why they're happening.
In some cases though you must read the docs carefully.
Im still wondered how people could give a green light to something that makes their life _harder_ not _better_ just because “it’s Google”.
My guess is — you can make more money on something as overcomplicated and non-developer friendly as k8s.
Imagine how many companies would never even exist if k8s was like a bit advanced version of swarm.
How many talented developers would be free to make something valuable for the world.
Instead of just solving stupid problems created by another developers with questionable design choices.
Honestly, I'm not sure Kubernetes is it. K8s seems like a leaky implementation detail when what people want is just to virtualize a workload (with similar simplicity to Docker) and have it work wherever the standard 'workload virtualiser' runs.
Perhaps the enterprise class stuff is always just for large division of labor efforts. Like the "Liberty" stuff from WebSphere perhaps this really just saying if smaller groups and less people want to mess with it, it should have some new thought around more consolidated abstractions.
I enjoy using it and playing with it, but so many use cases can be addressed with something simpler - either just Docker / Swarm / AWS ECS etc. alternatives or just going for VMs with well defined CI/CD processes that let you tear down the infrastructure and set it up again easily.
What very interests me are the concepts K8s build on that are not usually recognized - to me K8s seems a lot like a JVM, just that it operates on infrastructural (and not runtime) level.
I enjoy experimenting with these concepts when applied back in the runtime world - it is for instance interesting to run 100s of servers with JVM and let them load/execute new dependencies and code at runtime (JVM is very well suited for that).
This is area that is not yet explored and would probably deserve more attention as it allows for distributed rapid computing that is infrastructure/platform independent (the downside is that it requires (just) JVM and the isolation is not perfect).
Now, can you get along wit K8s? Of course you can! Human beings have been adapting to harsh environments for hundreds of thousands of years. We can figure out some shitty software. So that means that probably, K8s will not actually improve, because people will probably adapt to it rather than change it.
So I think it's time to create a new distributed computing platform. Something with a simpler design, that has the necessary functionality built in rather than bolted on. Something that is very powerful but also terse. And of course, something that only gets complicated when you need to do something complicated, and allows you stay simple as long as possible. The ability to scale the complexity, essentially.
I'm learning Kubernetes and is deploying my own test cluster on my ARM-based board at this very moment, and I already spend 3 days on K3s and have to give up due to a problem (https://github.com/k3s-io/k3s/issues/2509#issuecomment-78657...). I must say, this is way way harder than Docker Swarm.
Just want to know, will Google actually make Kubernetes a bit more friendlier for general non-GKE users?
Google actually works a huge amount with the community to simplify Kubernetes across a number of SIGs. Its always a trade of increased flexibility and options as people use it for more workloads, versus simplicity.
Autopilot is just for GKE. You can use GKE on other clouds and also onprem (bare metal or VMs with Anthos)
What I miss the most is having my infrastructure defined as code, instead of via the GUI. But given that I have only four services (out of which two use preemptible VMs and only one needs to scale) it’s not really a problem — it wouldn’t take me many minutes to replicate this setup at another cloud provider.
Dedicated servers are heavily under utilized and over provisioned because once you allot a server to a team they don't want to give it back.
VM solved this problem and changed how servers were provisioned. Docker and K8s are the next progression of this. People who compare K8s to the 'next js framework' have to do some serious context alignment...
I really like "Google Cloud Run" in this sense. Just deploy your Dockerfile.. And you're done.
A single yaml file. Limited configuration. Scaling just works. No cluster management.
I just want to build a great product.
Is a tool that will allow the configuration of base images. It is actually a thin layer wrapper around packer and ansible
Of course it's complex, it's managing a complex and very deep problem space through a mostly consistent generalized interface. Whether it is better than the alternatives depends on how deep your problems in that space go.
They should remove the need for CONFIGURING ssh.
Now they have removed the entire control plane of the container. Hiw should a developer debug something then?
You can still shell into your own containers to debug workloads, we removed SSH access to the nodes in order to provide management of the nodes (we can't take on the management if people can SSH in and make unsupported changes).
If I enter the following, it seems to work for me: Replicas: 1, CPU: 0.25, Memory: 500MiB, Ephemeral Storage: 1 GiB.
The result is $9.92 per month for the us-central-1 location.
The issue here is we don't have a kernel and compiler for distributed applications. So instead we have no choice but to expose that complexity to developers. They need to specify the IP address and port of a remote service to invoke. We've solved the IP problem in part with DNS, but now what happens when you want to migrate or load balance? Kubernetes solves this with service discovery, so you just name a service and the container orchestration engine worries about resolving that to a pod at runtime. But you still need to specify a port. We haven't yet figured out how to automate allocating ports like we managed to automate register allocation on a CPU, especially when application code itself might need to know them. We still require you to know how much storage you need and specify that in the definition of a persistent volume. Ideally, it would be as easy as it is to ask for an array of integers in a programming language. The compiler will figure out how much storage an application requires and give you that much from some pre-allocated pool you as a developer don't have to worry about.
But again, there is no such pre-allocated storage pool. There is no such compiler. There is no POSIX filesystem or memory standard for multi-machine networked systems. There's a fragmented system of vendor-locked services providing storage servers, database servers, cache servers, http servers, message queues, some more open than others. Kubernetes is an attempt to provide abstractions that make it possible to define an entire network of such individual servers declaratively and it's a noble effort. But it's complex because the underlying problem space is complex. Distributed computing is at the point in its evolution right now that single-machine computing was in around 1950 or so, when you needed to tell the program exactly where in memory to find and store a variable, exactly where on disk to fetch a block of bytes. Will it ever get to where we are now with device drivers, compilers, and kernel allocators and schedulers doing all the heavy lifting for you? This is what a Kubernetes engine is trying to be, but it's early in the game. Very early. I don't see the point in writing an article implicitly shaming the developers for trying to provide higher level abstractions that make the simple cases easier, any more than criticizing a kernel developer in 1960 for inventing virtual memory as if they're admitting that symbol to address resolution in a multi-processing system is too complex. Of course it's too complex! And we're trying to make it less complex.
We are currently on AWS ECS which works very well for us but need to move to a GDPR/Privacy Shield compliant environment hosted in the EU. I was to look into Managed K8s but am turned off by this thread :-)