Errors from Microsoft's Cloud | Hacker News Reader

Errors from Microsoft's Cloud

errorsazurethrows.tumblr.com

139 pointsdonselaar57 comments

I know multiple customers with significant free Azure credit that prefer to pay for Google Cloud or AWS over using Azure for anything critical.

The management system is a mess - the new portal is nutty and slow. The old one lacks key features and operations are still slow. I can't understate how frustrating and time consuming this is. Maybe I'm using it wrong, but whatever. Using GCP is a blast of fresh air - light, fast, straightforward.

VMs are way overpriced and have poor performance. Windows boxes not-so-infrequently get stuck during reboot, and we were told to resize them to get them going again. "Premium" storage aka SSDs have a terrible design. The temp disk system is nonsensical. It just goes on and on.

It's obvious Azure is a great play for MS. They need to convert their enterprise customers that were spending tons on MS before to spending tons on MS online. And I'm sure they'll do fine there (recent earnings confirm?).

I had an apparent MS employee reply to one of my comments complaining about azure's portal saying the engineers know the interface is awful but can't do anything about it:

https://news.ycombinator.com/item?id=12773022

I'd assumed it was a design by programmer fail, especially because of the terrible modular design, but it doesn't look like it.

I was wondering, poor performance wise, a client of mine is consistently seeing a 3-5ms lag between the SQL boxes and the website on azure. This seems incredibly high to me, but my experience is generally with SQL server on the same box or on the local network. I've never had a problem like this before, but this site can generate hundreds of queries per request because of legacy code.

That basically adds 500ms to a request just going up and down to SQL.

Am I being crazy for thinking a 5ms latency on stuff that's supposed to be in the same data centre is incredibly bad?

We're also getting random "The wait operation timed out" on DB connections between azure SQL dbs and Azure websites, basically a complete DB hang that I've never seen on a dedicated server setup, the client upgraded to premium, they're getting them even more. That means random users are getting complete failures every now and again. Unbelievable.

I honestly can't understand why anyone would pay for azure, in my experience the performance is an order of magnitude worse than dedicated servers that are half the price. The only good thing about it is easy deployment, which is something that's a complete PITA on dedicated servers until you've spent a bunch of time setting it up.

I can understand the comment "UI by designers", it looks pretty when you first logon, but then you try and use it then it feels slow, and quite an effort to get around.

I think there are actually three types of UI designs.

1) "UI by programmers". Very low use of graphics and animations. Everything looks very standardized. Usually has advanced options panels that let you tweak small parts of the algorithm. Gives a high level of control as there is little abstraction between the functional and presentation layers.

2) "UI by designers". Use of graphics and animations where necessary to make the UI more understandable, such as animations to show the result of user actions. Code is usually abstracted away and the interface is set up with the user in mind. Steps to perform common actions are very quick with the minimal amount of user input.

3) "UI by artists". Excessive use of graphics and animations. Animations that serve no functional purpose. High use of art with large images used for theme rather than function. Often no real link between what you expect buttons to do and what they really do. Steps to perform common actions are usually convoluted with multiple menus. Some even use hidden menus that must be invoked through arcane gestures in an effort to make screenshots look more appealing.

I think the UI being described here is actually the "UI by artists" rather than a well designed UI by a designer and not an artist.

I agree, although on first glance the UI of azure's portal seems to be a 1 type UI.

I think a programmer UI would be fast, with a lot shortcuts and easy get around ONCE you know it. But awful for non technical people.

You aren't crazy. I'd recommend testing from your app with a simple speed-of-light test (e.g., SELECT 1;) just in case your queries are slow.

Perhaps Azure SQL did the same thing we sadly did with the first version of Cloud SQL: run it in a totally different environment that requires you to go through several hops to reach the actual (MySQL in our case) server. But even then Cloud SQL First Generation was "only" 1ms away. Second Generation which runs in a VM but managed by Google (like RDS at AWS) is basically equivalent to running it on your own VM, and is thus about 100 microseconds away.

Disclosure: I work on Google Cloud.

PS. We recently launched SQL server images on GCE (https://cloud.google.com/sql-server/) though it's not a managed service (yet?).

Yeah, that's how I figured out the cause of the problem. Had fixed a load of performance problems on a page, but couldn't get rid of 500ms compared to my laptop and realised it must be query latency and was surprised to see 100 "SELECT 1"s using the same connection consistently took 300-600ms (and it jumps around all the time in that range). The ridiculous thing is that the test server running on S1 has better performance (but still has a 200ms query lag for those 100).

And that's leaving aside that heavier queries take 6 times as long on a so called "Premium" instance compared to my 2 year old laptop. Part of me suspects one of my cheap VMs running SQL Express could do better than azure's "premium" db performance.

I also know it's not because of DTUs (MS's database usage metric), my client's site is seasonal and is just ramping up to prime-time, it had the same crappy latency in July when it was handling 1/10th the traffic as it has now.

SQL Express has several performance limits and goes idle when not used.

If one has free credit for cloud resources, why can't they use the free money to just spin up some resources for non-critical work? Say log processing, run public tests etc.... I know I would if I were running a company and MS comes up to me with this.

What if the hassle simply does not outweigh the cost of the flaky VMs?

There are several reasons the hassles might not be worthwhile even if the free VM is 100% reliable. Most other VM hosts offer mostly Linux solutions and Linux skill is not always transferable to windows. If your organization has those skills then the wasted time cost might than a familiar VM would have cost.

You can run linux on azure.

I've chatted with people who've gone through Microsoft Accelerator, and that's exactly what they're doing.

Yep. Also I can confirm that for at least some accounting, MS treats the free credits as revenue. They will give discounts and consideration based on your spend even when the spend is just credit another dept gave out. Clients have told me of very happy calls from MS reps when their projected spend for the year exceeded 50K even though it was nearly 100% credits.

If you're going to do log processing, it means you've got data to process. And I haven't worked with AWS since 3 years ago, but in general external bandwidth is expensive, which is why you want to keep everything in the same data center where bandwidth is free and where you have no latency.

Good point. Agreed.

But, don't the free credit cover the bill incurred from bandwidth as well?

There are a few issues, but to be fair I've encountered similar things in all the cloud and VPS hosting services I've used. However, I do remember that I was once trying to push Azure as a solution and they promptly had an outage. We went with AWS instead. It's not that AWS don't have outages, they do, just that the timing was really bad.

Also, don't use the preview services in Azure apart from to try them out. They really are previews.

I've recently written a three part series of blog posts on AWS vs Azure. The third part, on pricing, is out today.

https://unop.uk/on-aws-vs-azure-vendor-lock-in-and-pricing-c...

Edit: One plus of Azure (at least for me) is that it runs on renewable energy. https://www.microsoft.com/about/csr/environment/renewable_en...

To illustrate the "don't use previews" point, they recently announced VNet peering where two VNets could be peered together without VPN gateways.

We started using them and at some point discovered that half of our instances in a VMSS could not talk to the other VNet and half could. We were tearing our hair out, there is absolutely no configuration difference between those hosts.

We did contact their support and after a few days they came back with this: "there is a small bug with the peering that we are aware of, only hosts which were launched before peering was set up can communicate with the other VNet, for new hosts to communicate you need to delete the VNets, just recreating the peering is not enough"

They did offer to add our hosts in by hand, but we were running back to VPN Gateways ..

... which also have a peculiar problem, the Standard level GW's are capped to 100Mbps but burst to 200Mbps and then get throttled to 0 for about 20 seconds or so. This was also confirmed by Support.

Our experience with Azure has been very painful, things look good on paper but it's pretty much lies or illusions.

In the month we trailed it we found there were so many internal networking and performance problems. I went with with a suspiciously optimistic attitude as I am experienced systems delivery engineer and I don't find windows especially in the server special at all reverent these years, but I tried to objectify Jude it as a platform. Ill tell you what - it's built to make money for the backers - it's not built out of passion for quality or creativity. The amount of unexplained outages they deny or avoid with low even phone support desks. I would estimate that approximately 90% of the outages we noticed (all platform / service related in the Asia/Pac regions) would never appear on their service status pages. We also have been trialling office365 which runs on the platform. We measured from 2015-2016 we only had 94% availability of service with about a 1-3%/100% margin of error due to monitoring resolution etc.... data corruption, internal CA and NTP problems clearly show from the underlying PaaS, you see things bouncing around a bunch of badly related windows servers barely grasping the idea of modern virtualisation and application abstraction and purely built to make the fat fatter and as I said not out of passion.

objectify Jude -> objectively judge? :)

> I would estimate that approximately 90% of the outages we noticed (all platform / service related in the Asia/Pac regions) would never appear on their service status pages.

Sounds kinda par for the course. At my last job we joked that the AWS status page would only be updated when the datacenter was literally on fire.

Red on the AWS status board means there has been an extinction level event, right?

Maybe that would still be a yellow - intelligent life could evolve again and resolve the outage.

Some of these seem a bit unfair to Azure, but this one was pretty amusing:

  User Impact: Users may be unable to activate or use
  SharePoint Online Sandbox Solutions. This may include
  event receivers, workflows, web parts, feature receivers,
  and InfoPath forms.

  Scope of Impact: While we're not aware of any customer
  reports of impact, our monitoring indicates that any
  user attempting to activate or use this feature will be
  affected.

Doh!

Consistent with most other SharePoint things that I'm familiar with. ;)

Some of them seem extremely misleading. The one claiming that Exchange was down for 20 days is politician-level dishonesty. The service wasn't even down. A small set of customers experienced repeated notifications for meetings, which is undoubtedly very annoying but nowhere near the impact of having no email access for weeks.

Most of these seem pretty minor. It almost feels like the complaint is that Azure is being too transparent. If Azure hid the issues until you encountered them directly, you'd not even see most of these. Yes, they should fix (and presumably are fixing) these issues, but transparency is a good thing for both the team and the customers.

Disclosure: Microsoft employee

I agree (with Azure not being ready for primetime).

Their subscription page is incredibly slow: I timed it taking 17 seconds to load! http://forwardscattering.org/post/41

After using some Azure VMs, I found there was still some storage devices or something on my account, several hundred of them. I tried to delete them through the Azure website, but you can only do it one at a time, and each deletion requires several clicks!

So I gave up and tried to use their powershell interface to delete them, but kept getting complicated errors. Eventually I decided f*ck this, and just deleted my Azure account. They're still sending me emails.

Normally you just delete the resource group they are in.

Oh yes, loading Account page to check the latest invoice takes ages! Good thing i have to do just once per month.

Portal is getting better tough, I can see changes every time i login. But still I'm getting Guids as error messages, thats still there

The localization one, with all of the languages, is that really a bad thing to do on an error page?

Apple does the same thing in some of their UIs.

If it's a brand new experience (like a user manual, or first logon), or you're in an error state - might you not know the locale of the user, or have failed to load the correct es-MX.json strings file or whatever it's called?

Also displaying error in multiple languages is clearly better than displaying only localized one because localized message can be ungoogleable and unbingable.

Some of those are scary. Our team runs everything on Azure, for over 2 years now, and I've only seen one outage that lasted maybe 30 minutes. Maybe we're lucky to be running in a good region, or not at some massive scale. Our biggest issues come from figuring out edge cases with new features that aren't well documented yet. That and the cost of using some features like Machine Learning makes it unfeasible for a team company our size. Azure Functions and API Management is sweet though.

I haven't tried many of Azure's more advanced features, but I've found the admin interface to be a pain. The series of UWP/Metro style panels and flat design has been a bit clunky and non-intuitive when I've had to use it.

I don't intend that as a slight against anyone who works on the Azure admin UI. I realize that trying to create an interface to a complex set of services is no easy task.

The old management UI is better, IMO. I get lost in the never-ending accordions of blade panels in the new one.

Oh man, so true.

Makes you wonder if this is anecdotal or rather the common experience for most of the users with a pretty big stack on there? I've used Azure solely for running like the cheapest plan for an Ubuntu VM and it has been flawless for that.

Another anecdote: I manage a a couple of hundred VMs, databases, storage accounts, event hubs, Azure Web Apps and much more. Errors in the UI is pretty much a non-issue to me. I have multiple user accounts and manage multiple subscriptions and sometimes switching accounts gives me an error and I have to logout and back in. A bit annoying but hardly something which would make me switch.

I do suspect that their portal will not work well if the connection between you and their data center is shaky though.

I tried to use Azure and found it to be a complete non-starter.

Stay away from this mess if you value both your sanity and productivity. Microsoft is still raking in the money from big companies who have people working for them who LOVE to work on these kinds of things. It distracts them from actually getting any real work done.

You know, the kinds of people who love to say "I'm blocked" at their standups.

Azure isn't ready because it throws errors sometimes? Welcome to the cloud. I only have limited experience with Google Cloud, but AWS has its share of issues like any other piece of software.

With Azure at least MS appears to be attempting some useful error messaging.

I think the issue is more than the tooling randomly spills its spaghetti, rather than some percentage of instances failing, which is obviously expected.

Personally some of those screenshots would cause me to run a mile if I ever encountered them in my work. Those kind of generic, unspecified errors are terrifying when dealing with other peoples code which is ostensibly stable.

Which one would make you run a mile?

The 'undefined' error in the CLI tool, the generic "this operation failed" messages in the dashboard, the unexplained authentication failures. If I encountered those multiple times when trialing a service like this, I would abandon it.

It's one thing for there to be errors in OSS code that I can fix, but with a huge and complex service stack like this you're basically at its mercy, and generic errors and failures leave you completely powerless. When something goes wrong with AWS, I've always gotten extensive information which aided in debugging.

I guess if you're part of a large enterprise you have a support contract with MS which gives you leverage around issues like this, but that doesn't apply at the level I currently work at.

Azure has some pretty neat tech but their UI and CLI tools are the worst of all cloud platforms (and I've used everything from AWS, GCP to DO and Softlayer). They're always slow, confusing and hard to use.

Considering this is how you interact with the platform, I'm always surprised why they don't devote more of their considerable resources to just fixing this. It can't take more than a few weeks to revamp the whole thing. Seems more like political and management problems on their end than any tech issues.

If you tried using Azure you were met with two different web interfaces who are mutually exclusive (by this I mean you can't manage resources created in one via the other) plus MANY CLI tools with overlapping capabilities.. Finding the relevant documentation can be a pain at the least.. We were trialling it as we got it for free, and got off of it as soon as we got the AWS free start-up package.

The 404 error page that was actually localized to several languages looked pretty useful. I don't understand the problem with that page.

What if you speak Xhosa ? Computer says no. It's about (very probably) knowing the user`s internationalization / locale and still displaying 50 other languages they (very probably) don't need to know about.

Localized means one message in your language.

These are mostly issues with the two web portals. The portal is awkward and slow, and doesn't have a table view of my stuff. Azure has been fine for us for provisioning vms via pythons scripting. Of course, the API suffers the same problems as the portals, there are two, and it's kind of awkward and slow. Neither is it well documented. (I had to use an undocumented feature for months for something basic... Might have been auth with the certificate. I'd also say azure is the slowest out of it, gce, and aws for provisioning a VM and installing a bunch of crap. I don't have real data, just been my impression comparing similar cost plans. That said, it does run Linux VMs.

The main shocking thing for me was learning that for all of these cloud APIs the calls fail with regularity, and you can waste a lot of time retrying shrugging "we dunno what happened" server side API problems.

I used to use Azure as Digital Ocean++ (As a student and a MSP , I got around ~$200/m credits to play around with on it) and it worked great.

However, after their facelift (the new portal), they are trying to be more like AWS and I found it significantly more complicated to do simple stuff (like adding ports, or load balancing) that was much easier before. Furthermore, I find AWS easier to use (maybe because of it being more popular and having more people blogging about it) compared to Azure.

The point of this comment is...Azure is an awesome platform if used as a Digital Ocean++ for now (and if you can afford it), otherwise I have personally found AWS and GC much easier to use (and cheaper in some cases).

Disclosure: I used to work as an MS Evangelist & was an MSP.

Most annoying thing - is that Azure unable to transfer account subscriptions region.

I've relocated from one country to another, and only option I have - is to create new Microsoft account with different not seen by MS credit card.

Only credit cards? They accept debit cards, right?

With apologies to the anonymous wag(s) who long ago composed Windows desktop error messages in haiku format [1]:

  Windows Azure crashed.
  I am the Blue Screen of Death.
  No one hears your screams.

In fairness, I should note that I've personally had no problems with Microsoft cloud services.

[1] http://soler7.com/IFAQ/WindowsErrorHaiku.html

We're looking into making a migration to Azure.

I was hoping to get insights from HN on the following:

- Using Azure CLI instead of Portal UI for standing up infrastructure.

- General DevOps tooling support on Azure (Terraform, SaltStack etc...)

- Reliability of Azure in general.

Azure seems to be moving fast to catch up to GCP and AWS so we're optimistic about it in the upcoming years.

Must say I disagree with most of the comments here. I use both Azure and AWS and would choose Azure every day of the week. The Azure portal UX does look like it was designed by designers, but the AWS portal feels like it was designed by developers, totally unintuitive. Something in the middle would be better, like the old Azure Management.

My favorite is if I try to login to my Office 365 account from https://products.office.com and "sucessfully" login then every page on that subdomain starts giving Runtime Error.

Yes. This. My own horror story: http://www.rollc.at/tech/azure-no.html

I tried to sign my son up for Xbox Live from several devices over several days, and all I ever got was a generic error page.

Can't zoom in on the images on iPhone for some reason.

Reply on news.ycombinator.com