We used our own product to learn about and debug the issue. Its rather wild that they'd roll out this change so incrementally, which my colleague outlines here.
Gradual rollouts are pretty typical to give the team a chance to do a rollback before they cause a complete outage. This particular usage pattern probably just didn't appear as a significant enough spike in error rates.
Ya, that makes sense, it really isn't a normal use-case. I wish we kept tracking the other regions to see if they have had this change roll out to them yet, or if it's still slow rolling.
Is a global outage better than a regional one? You can’t predict the impact of changes on a system as complex these.