Lack of backpressure is perhaps the most common newbie mistake in distributed systems. I've just seen so many failures because the servers were overwhelmed and the clients were greedy/impatient (didn't react well to anything less than an immediate response even when they were doing everything they could to overwhelm). It should be a primary design consideration for any such system being built anew.

> Backpressure Strategies: control, buffer, drop

One of my favorite hacks for Gluster (at Facebook) was in this area. We were using bog-standard NFS as the connection protocol, so neither explicit control nor dropping were options. (People who have always had that luxury with their own thrift/grpc/whatever protocols and libraries don't realize how much of a pain it is not to have it.) That left buffering, which led to memory pressure and latency bubbles. What I ended up implementing was implicit control. We'd just stop reading on a particular client's socket for increasing numbers of milliseconds if overall latency was too high and its queue was too long. This leveraged information we already had, and "borrowed" TCP's flow/buffer control as our own. Clients couldn't ignore it; they'd block in send() or somewhere equivalent. It worked really well to smooth out many of the worst latency spikes. If you're ever stuck in a similar situation, I highly recommend this approach.