No, you would not want to run a cluster per service; Kubernetes is somewhat multitenant. You'd get better efficiency from running one cluster per datacenter, with all those teams in their own namespaces. Kubernetes can stack many containers per physical host, with widely varied workloads cooperatively sharing.
Until you learn the downsides of this approach at hyper scale, namely that containers and a shared kernel mean that all of your workloads are sharing the same kernel parameters, including things like network limits and timeouts and file handle limits. Multitenancy and containers actually ends up working against you and creates new problems and knobs in your individual jobs that you have to configure -- to the point that it's almost worth just having different types of jobs run on different isolated node pools and eliminating your multitenancy issue anyway.
Companies that scaled on KVM never had to learn about these limitations and just focused on what their hardware was capable of in aggregate.
At hyper scale and with multitenancy, microVMs are always going to be the end state -- and while there's k8s support for this, it's far from the default or even most convenient option.
File handle limits are something no sane workload ever encounters. They are technically a shared resource, but in a sensible kubernetes configuration, it is impossible to hit because the ulimits on each process are low enough. A very small number of teams may need an exception, with good reason, and will typically be cordoned on to their own node classes that are specially tainted.
Yes, fleet Management via taints offers nothing over the fleet Management that you've already got. This is a good thing. Fleet Management tools are a damage to your reliability. They mean that your machines are non-fungible. Kubernetes great innovation is making machines, units of compute, fungible.
There are workloads and architectures that will never be suitable for kubernetes. There are HPC Clusters that heavily rely on things like rack-locality that Kubernetes views as damage. Getting rid of them is a net win for humanity.
[1] https://kubernetes.io/docs/concepts/configuration/manage-res...
I guess databases are not a sane resource? I've seen file handle limits hit with databases more than once in my life, and that isn't specific to k8s.
High performance template rendering has as many filehandles as open requests - Maybe 10,000. If it's actually high performance, the templates underneath aren't files anymore by the time you're processing, they're stored in memory.
Databases are almost an exception, but you shouldn't be running "Large DB" on a shared host on K8s. You should taint and dedicate those machines. K8s is still useful as a common management plane, but I'm roughly on the fence of "Just run those machines as a special tier" and "Run them on k8s with dedicated taints", because both have advantages. Smaller databases run just fine. Postgres is using ~10k filehandles
There was a time for a scheduling specifically for "filehandleful" jobs. It's long gone. Modern linux systems set the filehandle limit to something obscene, because it's no longer a limiting factor, and it hasn't been on these workloads for 5 years.
"Multitenancy" here means "I have a giant pool of machines and I run a bunch of jobs across them" not "I have a pool of giant machines and I stack jobs on them".
Instead of this you can have central infrastructure and platform teams have partial ownership of infrastructure while service teams or departments have partial ownership. In that model a service team or department owns their own cluster with infrastructure code partially provided by the platform team running on compute infra maintained by the infrastructure team.
This model has seen much success at the scale of Amazon while maintaining SLAs and controlling costs. There are of course a number of drawbacks to this model at scale that you can ask any former Amazon engineer about.
I would think a similar model can be mapped to hyperscaling kubernetes where operators, cross cluster infra, and base kubernetes configs are maintained by the platform team while departments or service teams (depending on scale of team size) maintain their own clusters at whatever granularity fits the company’s scale (e.g region, datacenter).
This is also where cloud can help alleviate some pain for your platform and infra teams by using managed solutions to solve some of these problems.
In my opinion both are viable models at scale it just depends on the needs specific to your company.
The biggest one is just mechanical: with that many clusters it will be hard to move capacity between clusters, and locality gets baked into everything you do (people do try to build around this, but it's awkward).
If each service or team runs their own kubernetes that's a lot of overhead: kubernetes will need something like 6-7 machines for the cluster (I don't have production experience with kube, spitballing here), so small teams or jobs will have terrible efficiency. Big teams will have to spend a lot of operational effort to manage their fleets.
It's worth noting that at hyperscale there will be individual jobs in a datacenter that are bigger than kubernetes handles comfortably. Handling this efficiently becomes very important, it's literally billions of dollars worth of hardware.
While I loathe the term hyper scale... even a single AZ in a single region could be 50,000 machines. k8s is a dumpster fire at that size.