Unavailable servers continued to retry requests for membership data, maintaining high load on the metadata service.
It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
It sounds like the retries exacerbated the situation. I wonder if they use exponential backoff?
It seems that perhaps more time/effort needs to be spent testing the systems (and the use of those systems) which are critical for handling failure. While building them, it's easy to dismiss their scaling requirements on the basis that they shouldn't ever be under significant load.
More importantly, did they use randomised exponential backoff?
Having all the retires hitting at the same time can lead to a pulses of outages until things settle down.
I'd say the Dynamo team are well aware of what they should have been doing, and kicking themselves for not foreseeing this cascading-failure case. ouch!
My guess is that a case like this you'd hit the backoff's upper bound pretty quickly, and, given the large volume of servers hitting a comparatively small metadata pool, experience exactly the same failure mode.