Look into kops as well - great setup tool for well designed deployments.
For instance, if we see that our worker cluster is has reached some consumed capacity threshold (CPU, memory, etc) then we need to add more worker nodes. The way we do this is via Auto Scaling Groups. Auto Scaling Groups work on launch configs that are essentially an AMI and instance user data. Getting an AMI that just has the Kubernetes stuff that can be started at will and told to join clusters at runtime has not been clear from the documentation I've read.
When used with Kops, it should give you what you're looking for (with the baked in kops AMI)
- It only scales worker nodes
- It only scales nodes if a pod can't be scheduled. This is too late for us as are requirements are to scale when resource utilization crosses a threshold (ie reserved memory is over 70%).
- It still doesn't solve the baked AMI problem. I may be wrong about this but the AWS docs for CA are lacking. For instance, when does the ASG/launch config get created? Is that something it does or it needs to be done ahead of time? Which AMI does it use? etc.
1) Yes, but I'm not sure why you would want to scale other nodes (I assume you're referring to the masters?)
2) Yes, the cluster-autoscaler will give you more node capacity. If you want to scale Pods on a CPU basis, you can use a Horizontal Pod Autoscaler (https://kubernetes.io/docs/tasks/run-application/horizontal-...). I believe it now support custom metrics as well, so you can scale on any resource threshold.
3) The ASG/Launch Config is created by Kops when you create the cluster; and is also manageable through the kops tools. It will default to the kops default AMI (which includes the kubelet, and everything you need for k8s to run), but you can also override that with a different AMI if you have a k8s compatible AMI that more fits your needs.
A contrived example is imagine if for every gmail user that logs into gmail google spins up a pod for the user. No imagine if there wasn't any capacity to schedule a pod for a user that is logging in. That user would either be denied or have to wait for an instance to come up and then the pod to be scheduled. Not ideal.
[1] https://github.com/cloudfoundry-incubator/kubo-deployment
Disclosure: I work for Pivotal, we built Kubo with Google.
This guide is more of core recipe from which someone can bake automation with config management or custom images.
We built a Kubernetes platform installer[1] using Terraform which works fairly well and I think is more white box because it uses Terraform directly. However, we are still dealing with some of the necessary Terraform language repetition in our modules.