- The problem with bind before connect is that the OS thinks you might call listen after bind instead of connect and listen requires the src port/ip to be unique.
- Running out of ports may affect short-lived connections too: by default when you (the client) close a connection, it goes into a ‘TIME_WAIT’ state that locks up the {src,dst}_{ip,port} quadruple for (by default) 2 minutes to protect new connections with that quadruple getting packets that were meant for the previous connection.
I think one thing that weakly surprised me about the article was that cloud flare seem to be mostly using standard BSD sockets apis and, presumably, the kernel interface to the network cards. I would have expected their main products (ie being a CDN) to use more specialised network cards and userspace networking (eg the final function they give for UDP does a ton of syscalls, and for tcp it still does several) with a non-BSD api. The main advantage would be having more control over their system and the ability to avoid a lot of unnecessary overhead from syscalls or apis, but there could be other advantages like NIC TLS (though this can also be accessed through the kernel as kernel TLS).
I’m sure cloudflare have reasons for it though. Maybe the hardware is more expensive than underutilisation or hard to source or buggy or user space networking just doesn’t improve things much. Many of those things can make servers more efficient but that might not be necessary if many servers are needed at lots of edge locations with each server not needing efficiency. Or maybe those things are mostly just saving mics when the latency over the wire is millis.