We did thought to use queue depth, but CPU metric was easier to start with. One problem with using queue depth though is that we can't (didn't figure out how to) throttle down autoscaler to not give us more machines in case downstream service is down.
We did upgrade Docker, but we didn't upgrade kernel. Switching kernel is a bigger task that we'll probably do at some later point.