I played around with a go server to do some simple scaling numbers - looking at possibly using go to implement a large-number-of-idle-connections notification server.

I found the (good) result that I could spawn a new goroutine for each incoming connection with minimal (~4k) overhead. This is pretty much what you'd expect since a goro just needs a page for it's stack if it's doing no real work. I had something like 4 VMs each making ~30k conns (from one process) to the central go server with something like 120k conns.

I found one worrying oddity however. Resource usage would spike up on the server when I shut down my client connections (e.g. ctrl-C of a client proc with ~30k conns).

Reasoning about things a bit, I think this is due to the go runtime allocating an OS thread for each goro as it goes through the socket close() blocking call. I think it has to do this to maintain concurrency. So I end up with hundreds of OS threads (each only lives long enough to close(), but I'm doing a lot at the same time).

Can anyone comment:

- is this guess as to the problem likely to be correct?

- is this "thundering herd" a problem in practice?

- are there ways to avoid this? (Other than not using a goro-per-connection, which I think it the only idiomatic way to do it?)

My situation was artificial, but I could well imagine a case that losing, say a reverse proxy, could cause a large number of connections to suddenly want to close() and it would be a shame if that overwhelmed the server.