Comment by deathanatos | Hacker News Reader

Comment by deathanatos | Hacker News Reader

It is a huge pet peeve of mine that,

a. the status page update took as long as it did.

b. the status page still only claims that AAD is down. You're not "up" if a required dependency of yours is down, IMO, and most of the Azure services right now are severely degraded to the point of not being useful due to the AAD outage. Now, that would make your status page look bad, yes. (And I would clearly message on it that the other outages are fallout of AAD on that.)

c. I don't have any real expectation that we'll get a public PM about this. (And a PM must include sufficient information for me to understand what went wrong, and what's been done to prevent it going forward.)

I had been starting to wonder what sort of incident it would take for them to actually update the status page. I guess this is it. We've witnessed a number of outages in various services over the past few months, none of which made it to the status page. I pushed a support rep on the issue, and was told that they don't want to cause a panic.

I'm also here because my work is blocked by the inability to do anything on Azure right now.

I looked at their past status histories, and they have provided pretty detailed postmortems for much more minor incidents. I don't have any reason to believe that they won't do that this time. I'm new to Azure, so this kind of outage really looks bad in my mind, but it's par for the course with Cloud stuff. They'll fix it and there will be one less thing to break in the future.

As someone who has been using Azure for 6 months, it's fairly bad but most of the Azure pain is more death by 1000 cuts than large explosions.

Documentation that contains triple negatives, UI elements that have no consistency (clicking off a modal on one page does nothing while doing so on another closes it and clears your data), widgets and modals forgetting state when you navigate to different submenus.

The sort of unexciting stuff that by itself is ok but it slowly makes you descend into madness.

Just wait until you see their made up rules for using their Redis implementation. You have to roll your own Redis to do anything interesting.

Their managed PostgreSQL offering also has fun quirks, such as that you have to connect using `username@db_name`, but your actual username in the database is still just `username`, so any third party software needs to support using a different username to connect than it uses to perform user-related queries (they have some sort of application-level load balancer in front of it that uses `db_name` to route the connection).

This is ... something - wow - they couldn't just make the actual username "username@db_name"

Nope. Their load balancer seems to strip the `@db_name` from connections, and anyway some software doesn't like having an @ in Postgres usernames (which is probably also a sign it's vulnerable to SQL injections)

Oh that solution is a garbage fire.

We hit the 4tb storage limit that they didn't tell us about on it when the documentation said 16tb. They told us we were on the older storage tier and that we needed to rebuild a huge analytics database on our time because they couldn't migrate us to a backend storage tier. So we had a database server that was in read only mode until we could migrate off of it.

Then we had random restarts and outages with their proxy that handles all of the connections. Which is why you actually need the username@db-name, because without the @db-name it doesn't know where to route you, because the postgres connection string is actually just mapping to one IP that they manage for multiple databases!

Fun trick is you can make your url anything that maps to the IP they use for their proxy and it'll take you to whatever the @db-name is in your connection string.

Their solution to the random connections dropping for minutes at a time was to "implement retry logic" on our applications.

Our solution was to migrate away from it.

We were well into implementation of a SaaS that relied on PG and used their cloud offering before we found that the latency was just abysmal.

The best we could get was ~45ms using pgbench vs ~10ms local and ~18 using AWS Aurora.

Their solution, wait for the flexible server offering which is not available in AUS but when tested in another region showed ~30ms so still not fit for purpose.

We eventually had to switch to running PG in AKS which has its pros and cons.

The funniest thing was, running Azure VM to AWS PG was faster than running Azure VM to Azure PG.

Been an active user of Azure for almost five or six years now. Things were a lot more rough in then (especially if you'd seen the old portal), unlike what it is now.

People always talk about cloud as an entity which can/shouldn't have problems at all. When it's more about convenience then anything else. So I wouldn't worry too much about this if I were you.

I don't particularly expect the cloud to be problem free. I do expect it to be possible to, when an issue is caused by an outage or a service issue, get that escalated to actual engineers. That has not been easy, IMO. Even obvious stuff, like "we sent X request around Y time and got Z 500 Internal Server Error from service S" have required multiple round-trips.¹

Messaging around transient issues helps. E.g., AWS would often email us to tell us "hey, sorry, such & such VM is being hosed by the underlying hardware, it is migrating but you'll see a forced reboot". That's all I need to know: I expect some interruption to that portion of the system, and I expect to see it heal within a certain time frame, and if it does, great, no support ticket required.

Azure sort of has that with "Resource health"; but, e.g., one of our recent support tickets is that we had a VM reboot unexpectedly due to a resource health event with its disks. And then, a few hours later, another VM, same resource health event, same issue, same reboot. And then again, a few hours later. And that pattern, with no further communication, requires me to write the (obvious) ticket of, "So… what's happening?"

(That ticket ended with such and such service was experiencing an incident. Never made the status page. It did make the internal "Service alerts" page.)

Part of issuing a PM is also to help show & convince me, the customer, that I won't be filing support issues for the rest of my life. But this last 6 months has just felt like support ticket after support ticket, and it's sort of depressing, since I'd rather be coding. I do think, last spring or so, it was no nearly this bad.

¹Every place I've worked at… we page on 5xx codes sent to a client. If Azure is internally doing that, it sure is hard to tell from where I sit.

Bad status pages are a very bad sign in my opinion.

I had to use the IBM cloud for a project once upon a time and the status monitoring was almost always wrong ...

I really hope that as customers we can succeed in making "excellent status reporting" an absolute minimum requirement for cloud services. Observing the quality of the status page reporting history is one of the first things I do now when evaluating a new service -- and no I'm not just looking for a page that shows all greens -- I'm looking for a page that connects to some conceivably meaningful metric as well as detailed reports about major and minor outages ... something to prove that they actually even know if their service is working at a given point in time ...

The IBM cloud page was a page that tended to update in response to my own issue reports (usually if I also pointed out that nothing about the outage appeared on the status page ...). I don't think that's at all an acceptable standard ...

Are status pages the right answer though? They seem to be a failure all around, so let's find a better solution.

Honest question for the sake of discussion/learning: What impact would it have on you if (1) the status page was updated instantly and (2) All impacted capabilities were marked as down/degraded?

Reply on news.ycombinator.com