Great postmortem report. I am very surprised to see that the DB ops simply deleted the index without even considering the consequences. More importantly, why isn't the 2 changes part of a single migration file? For e.g. in Django, south migrations can have both remove, create indexes in one file and executed together. Also, like some of the guys mentioned, why is the update performed to your entire global cluster. Shouldn't it be incremental like for e.g. one availability zone at a time?