I moved from a single massive instance to a target pool behind a load balancer with health checks based on if there's available queue.
I have not worked out how to geographically load balance both of those from a single domain based on which you are closest too, but I want to see if these new smaller instances in load balanced target group pools can scale to take the load like a larger instance.
Namely, load balancing and scaling based on CPU is not a good metric, because new instances (which are still serving multiple users), will keep absorbing new users before the metric is triggered, and even when it is triggered, a new instance will take a while to spin up and build some browsers, so scaling lags too far behind load and the effect is existing instances get and stay overloaded.
So, I have an idea for a new autoscaling system that puts 1 tiny machine per user, but it will take some reconfiguring. So, in the meantime, I'm switching back to the old system (massive instances, vertical scale).
I'll do that now and the servers should switch to the new system in about 30 mins.