What's the best way to make Kubernetes the layer that isolates tenants in a multitenant platform?
1. Do you understand all the primitives and how they interact
2. Do your use cases require some or all of these primitives be bypassed (host network, root containers, custom extensions / operators), which may invalidate most security
3. Are the human costs to manage 1 and 2 less than the human/infra costs to manage multiple clusters?
Many applications have a few assumptions that violate “safety on a single machine” - databases use raw block devices, processes run as root by default. The harder it becomes to isolate an app on a single machine, the harder it is to isolate that workload on a cluster.
In addition, the moment you install something above kube (helm, ingress controllers, operators that can read or write secrets across namespaces) you start to blur the security boundaries at a cluster level and thus make it harder to reason about who can do what. It’s a lot like the Linux kernel / userspace separation - everything secure is in the kernel, but if a driver or subsystem has a vulnerability that tends to compromise the whole kernel.
I would suggest most people run multiple clusters until you get to the point where you have cluster sprawl (and are paying for a ton of redundant / idle infra), and then start either with an opinionated distro designed for picking all the defaults for you (OpenShift, a few of the other vendors) or hire someone who can design a top to bottom policy (and even then you’ll want multiple clusters for failure isolation).
There is a Kubernetes Working Group where the definition and implementation of soft multi-tenancy and any feedback is welcome.
To all your workloads.
Kubernetes was simply not designed as a multitenant system. Namespaces aren't the basis of multitenancy, for pete's sake, they're namespaces. Piling ever more extensions on top to try and plug all the holes just highlights it more completely.
The concept given above is where I expect Kubernetes will go, either explicitly (by creating a hard separation between scheduling and everything else) or by convention and tooling (we all adopt layers of indirection like Virtual Kubernetes). The latter seems most likely to me.
This will, as others are pointing out, only get more and more necessary as CRDs and Operators continue their long march to gather all the knowledge of your systems into their embrace. But it turns out that storing all the truths of dozens or hundreds of distinct systems, some of which conflict, many of which can't be fully walled off from others, means that your security posture may include a certain amount of humming loudly.
Something like Virtual Kubernetes splits the difference: you can retain Kubernetes-as-scheduler, reaping all the benefits accruing from workload consolidation. But you also get hard tenancy boundaries between workspaces, between Kubernetes masters being used as a skin over etcd.
Kubernetes is amazing. But this is one thing it got wrong. Just wrong and we'll all be wrestling with it for years. It would be like setting out to write a single-user OS, database or filesystem.
Disclosure: I work for Pivotal, we do some Kubernetes stuff, but these are personal views.
Ill add that there are few well established companies that have adopted this approach, AWS, Google and Azure to name a few.
Automating the provisioning of an entire new cluster is very easy, at least for some people with the required resources. But even for a small shop it is possible to automate it using stuff like Jenkins and Ansible. It is still relatively sophisticated stuff. It is not rocket science but not an elementary school project level either.
The aks-engine repo let you peep to what goes behind the scenes, but this is not the kind of beast you will grow at home - it is a public cloud provider with global scale, with the expected level of complexity and engineering. It is available on: https://github.com/Azure/aks-engine
This is doable through various provider/vendor-specific APIs. Pivotal's PKS has an API. GKE has one, EKS, AKS and so on have them too.
In the longer run there's meant to be a full Cluster API, which I presume will take over for that role: https://github.com/kubernetes-sigs/cluster-api
Cloud Foundry.
Pretty much everything you need to do today is done. Multitenancy was designed in from the start and is available in several levels of hardness (orgs/spaces gives you user-level multitenancy, isolation segments give you component-level multitenancy, multi-foundation gives you the equivalent of multi-cluster multitenancy).
As time goes on it will adopt more and more of Kubernetes, Istio and other systems as they emerge and become able to take over from its own original components. But you'll get the same experience throughout. And Kubernetes will converge upwards as people realise that "It Just Works" is actually, you know, a pretty nice developer experience.
But I am, having worked on and around Cloud Foundry for several years, hilariously biased.
- Since essentially no container platform was designed for hard multi-tenancy, use automation to stand up separate copies of a platform at scale, using a platform with low overhead in controller to worker nodes.
Consider what would be the “simplest thing that could possibly work” e.g. offering tenants an AWS autoscale group (with appropriate IAM account, user, role, security group perms) behind an AWS PrivateLink endpoint:
https://docs.aws.amazon.com/vpc/latest/userguide/endpoint-se...
I say that, because most of these platforms weren’t intended for and are not marketed for hard multi-tenancy, until their engineers started getting asked the question by enterprises attempting it for regulatory/compliance/security reasons.
> Cloud Foundry. Multitenancy was designed in from the start...
This comment gives me pause, as CloudFoundry’s security model, approach, and engineering lacked hardness as of 3 - 4 years ago. We explored the multi-tenancy design, and found no guarantees, or even strong assurances, were brought by the platform.
The larger issue was we found CF’s security engineering assertions rather overconfident (flavor of the comment above). Warranted relative to other solutions without security features, but not relative to being hardened enough to trust in the face of bad actors, particularly malicious insiders.
My opinion is also “biased”, in the sense that unless a platform is doing work outside of “the server”, I don’t believe it can offer hard multi-tenancy.
Here’s an example of work outside the server:
> unless a platform is doing work outside of “the server”, I don’t believe it can offer hard multi-tenancy.
I admire the work AWS and others are doing to harden the hardware base, but I'm not sure how software would contribute or not contribute to that line of defence.
2) Service mocrosegmentation
3) 2 but using kubernetes network policies
You can also schedule Pods on Nodes that are for a specific purpose (eg only used by a specific team or product) by using various scheduler stuff like taints and affinity/anti-affinity.
Or you can write a custom scheduler.
In general I feel like all the tools are there in Kubernetes to build a good multitenant experience but it's also just easier to run a number of clusters, especially if you're cloud hosting where the control plane is usually free and you pay only for the workers.
I would though ask, what is sufficient isolation?
Multi-tenant, single cluster is tricky. You either need node pools per tenant, at which point why not run one cluster per tenant, or you need really good isolation of the workloads from the nodes, which requires PodSecurityPolicy or similar to prevent a breakout to the underlying node.
Also RBAC can be tricky in a multi-tenant setup as you need to be very careful with anything that would allow breakout to the underlying node or any cluster level rights.
You don't want to be colocated with bad pods.
https://istio.io/blog/2018/soft-multitenancy/
That works well to isolate services/apps/teams from each other, but also allows services to be shared where required.
I think if you wanted true multitenancy it would be simpler to spin up a second cluster, especially if you are in gcp and controlling everything using a infrastructure as code solution such as terraform.
It has pods, pod security policies, network policies, maybe even hypervisor instead of Docker, etc.
It's pretty much built for multitenancy.