Also most tools that were build to manage k8s cluster were nice to deploy a new cluster, not so much to upgrade a cluster so you would have to create new clusters every time you wanted an upgrade. It can scale when clusters and blast radius is small but can be complicated when it involves contributions from n teams. For this reason when we were managing our own k8s cluster on prem, we were using kubespray which worked but upgrades were a multiple hours affair.