Killing Containers at Scale
blog.repl.it
blog.repl.it
Have you opened a bug report in Docker upstream to see if they can improve the situation (perhaps by putting networking cleanup jobs to a background goroutine)?
> Have you opened a bug report in Docker upstream to see if they can improve the situation (perhaps by putting networking cleanup jobs to a background goroutine)?
I have not yet, but I plan to! We figured we'd work around it ourselves instead of waiting for a potential fix upstream.
Those of us who need to get things done have taken to just working around it and moving on.
If this attitude has changed, it should be communicated publicly so people can reset and come back to Docker with such issues.
Could you not just send a signal to the running process that it’s going to die and to hang up or save state or whatever is needed? If an Ephemeral vm ends in a bad state does anyone care?
Sorry if this is obvious: why is the request proxied to a conman that is shutting down in the first place? Wouldn't it be more robust to work around this instead? Is it perhaps because there is no way to tell when the VM is shutting down?
One of the most important invariants that we have to maintain is that there is only 1 container running per repl at any given time. We could determine if the machine is shutting down and not proxy the connection, but we wouldn't have a place to proxy it to since we can't be sure that the existing container has finished shutting down. So either way we end up returning an error and the client has to wait until the old container has been destroyed so a new one can be spawned.
Perhaps when a host is preempted, instead of killing containers at all, you could just add an iptables rule to black-hole all network traffic for the host. Then they are as good as dead (and the host will be forcibly killed soon anyways).
But I still would encourage even a 1s “clean” shutdown. You don’t need to wait for any of this fancy cleanup, but it’s really nice to finish your writes.
Fun story: for Preemptible VMs we started (in Alpha / EAP) with no soft shutdown just to see whether people could handle it (so just immediate power off). Turns out, that if your box is running apt-get upgrade at the time or anything like that, you easily corrupt your boot disk. So, we struggled between “a few seconds” (5, 15) and the “about 30, which is how long a GCE instance takes to boot”. That’s how we ended up with 30: we wouldn’t more than double regular instance creation times at the tail. Nowadays we boot to ssh in 15 seconds!
If you don’t reuse your state, none of this matters. But I’d guess that even just getting to the point of RST’ing the the connections is valuable (so that the clients know to take action, rather than wait a while).
We’re looking to fix up the GKE graceful node shutdown, because it’s currently “racy” and doesn’t actually respect the grace period properly (system pods / processes can be shutdown before waiting for user pods, causing you to lose logging or say the kubelet).
You’re kind of racing against the clock though. You could instead have a “load balancing” style layer in front, so that while there is no usable session, at least the person’s connection is just “hanging”.
Feel free to send me some email, as we’re looking to make this experience better (both generally and in GKE, specifically).
This way your client waits at most X seconds before being reassigned warm conman