Do you use something like kops for setting up and maintaining your cluster?
Most of my direct experience with Kubernetes has been on GKE, but I have been meaning to work through https://github.com/kelseyhightower/kubernetes-the-hard-way to gain more appreciation for what is going on behind the scenes.