coreysa

Almost everyone has fully recovered at this point. If you are still seeing problems with your Virtual Machine after the incident earlier this week, we want to help you!! Please send mail to azcommsm@microsoft.com and email me directly at corey.sanders@microsoft.com.

Please send with high importance so it pops in our inbox and we will dig in.

Thanks. The communications and the service health dashboard are two areas that we are creating improvement plans from the learning of this event. For the dashboard, we do expect it to continue to run even through outages like this one, but we did encounter an issue with our fallback mechanism that we need to understand more deeply.

For general communications, we did most of our early communication on the event using twitter, announcing the incident and giving updates. We need to build a more formal multi-pronged approach to communicating, including faster responses in the MSDN forums and here in HN to make sure we are reaching as many of our customers and partners as possible. Thanks again for the feedback!!

Keith, you can find more details in the RCA that is published here: http://azure.microsoft.com/blog/2014/11/19/update-on-azure-s.... It is updated with more details on the flighting and issues we encountered.

Thanks. We are continuing to investigate this and driving needed improvements in our process and technology to avoid similar issues in the future.

First, I am really sorry about the impact the incident had on your service. Specifically, for your changing IP, are you currently using the reserved IP feature. You can reserve both your external IP and your internal IP.

External IP: http://msdn.microsoft.com/en-us/library/azure/dn690120.aspx

Internal IP: http://msdn.microsoft.com/en-us/library/azure/dn630228.aspx

Hey nnx, this is Corey from the Azure engineering team. We have a standard protocol in the team of applying production changes in incremental batches. Due to an operational error, this update was made across most regions in a short period of time. I really apologize for the disruption.

Hi, this is Corey Sanders, an engineer on the Azure compute team. Yes, our normal policy for updates is to roll them in incremental batches. In this case, due to an operational error, we did not apply the changes as per normal policy.