AWS IAM Outage
health.aws.amazon.com
health.aws.amazon.com
Nothing quite brings on teh suckz like all the auth servers dying at the same time. Guess how I learned that lesson?
The progression of: Centralized Admin is a great idea, think of all the work I'll save!
->Oh no! The central db is dead!
->Oookay... Now I have two for redundancy...
->Oh no! Split brain! In distributed systems hell!
->Ya know, I might as well just give up and accept that this'll fundamentally be a distributed system, and put in the time to really nail down change propagation instead of chasing after a quick fix.
-><surrounded by instances of distributed systems in various states of exploding, being on fire, being extinguished, and brought back online. A layer of formmer junior developer brain tissue coats everything. Money almost exhausted, 8x more hardware acquired than originally spec'd out, no longer able to relate to normal humans, horrified by any proposals involving clock rollback> This is fine.
...is a hell of a ride.