Cloud migrating production first in 12 hours or less
We have a b2b webapp product in cloud_provider_a which started having trouble serving requests, timing out.
My team (supporting another product in cloud_provider_b) was tasked with setting up a duplicate of prod in cloud_provider_b. I can't even remember why mgmt thought that would help.
By the end of the day, by hook or by crook the call was made to switch PRODUCTION traffic into a newly made stack in a different cloud than where that application was currently hosted. Despite protest from the engineering and QA teams.
Its all a blur now. Things went horribly wrong for two days. On the third day my team (with no prior knowledge of how this product works) found that the docker containers running the app had an nginx config with a low worker_connections setting.
We changed that one line and the app is back up. But the cost has been high, our customers had a far higher level of disruption these 3 days. Our engineers feel burnt out and disrespected, and our QA teams are appalled.
AMA I guess