Hacker News Reader

Comment by blantonl | Hacker News Reader

I took our Web sites offline (RadioReference.com / Broadcastify.com) so our back end processes (audio archiving, audio serving etc) don't crater.

Our Master MySQL server uses EBS (striped RAID across 4 EBS disks) and it is getting killed due to severely degraded EBS performance. There is definitely a major problem with EBS at this time.

View on HN

We are running our sites across three AZs, all in US-EAST. Our three web servers (one in each AZ) are not even able to load static test pages from their EBS volumes at this time. We also have a Percona MySQL cluster across three AZs and we are seeing very degraded perfomance in the EBS volumes on all three servers as well. It seems like this is impacting more than a single AZ from what I'm seeing.

Well, we've gone from "degraded performance" to completely losing our EBS instance store for our master MySQL server. If we don't see resolution shortly we might have to look at promoting one of our slaves and moving forward from there.

We ended up promoting a slave to master, and moving forward from there. Our master eventually returned to service, and we'll promote him tomorrow after things settle down.

Interesting to note that we issued a reboot on our Master and it went away and didn't return for over 45 minutes - we thought for sure it would have to be terminated. API calls and console access was severely restricted, so even launching in new AZ's was problematic.

That's you? I love listening to my local sheriff radio. Nicely done.

Reply on news.ycombinator.com