Hacker News Reader

Comment by altmind | Hacker News Reader

Million times this.

Its shocking how "elevated rate of errors for specific endpoint" in your cloud provider status page is actually amplified to be a soft-outage of your product when your writes to disk never return, your databases returning inconsistent data or your orchestration taking some drastic measures for the failing health check.

When you have a lot of components in your cloud mix, failure of one stage(network->balancing->quering->rendering->persistence) bring everything down.

if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999.

cloud providers can claim mountain-high availablity whereas users will never get their apps running with advertised reliability for now there is multiple subcomponents that can fail.

View on HN

The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust.

Uptime and error metrics are technical and should be kept away from managers.

Maybe it's time for a consumer watchdog group to step in and do their own reporting for services like this. Like https://www.isitdownrightnow.com/ but with sharper teeth.

I'm not sure how much time I have to participate but I wouldn't mind chipping in a bit on a co-op in this space.

But it might be easier to convince Is it Down Right Now to grow some fangs, or socialize the idea that it does (perception counts for a lot).

Or it's time to go back to one's own infrastructure and take one's own destiny into one's own hands, along with the responsibility.

For quite a while now I've been hoping that cloud tools will hit the turnkey solution point where anyone technically literate can manage a small private cloud. Baby pictures shouldn't be on Facebook. They should hosted on my machine, and replicated on Uncle Bill's and Aunt Sally's, and only family members have access to them.

But I think I'm going to be waiting a while.

in a way Scuttlebutt provides this. Distributed social network. No central storage of anything. You post an image, it's on your machine, and the machines you connect to that have chosen to "follow" you. So, your baby pictures are on your machine and "Uncle Bill's and Aunt Sally's"

Realistically they're also on a "Pub" server which is just another client with a known IP that facilitates transfer of data between peers.

you could take it another step and build a personal image gallery / sharing client on SSB as the protocol supports (and has) many different apps that run on it.

I can't remember ever seeing this work out well lol. Happy to be proven wrong one day, though.

I have my own datacenter. Works well and has for the past 20 years. Costs me peanuts because I know what I'm doing and how to do it. Will never be an Amazon or any other "cloud" provider's customer.

how do you address the central "something went boom" issue of the post. Like if a backhoe takes out the line to connection to the data center do you still have any "nines" ?

I have set up everything to be redundant.

You may have heard of a company called Amazon.

It appears I had forgotten about that back-story!

What thread do you think you are replying to?

> own infrastructure and take one's own destiny into one's own hands, along with the responsibility.

Amazon did this. It went pretty well. So well they decided to sell the results of the expertise.

This was extremely evident in the slow response and poor communication during the recent Salesforce outage.

> The fact that many status pages are updated manually and any incident disclosure need to get approval from management(aws?) does not add to the status page trust.

Automated status page updates can also reduce trust, since then the status page is itself exposed to more kinds of system failures.

I worked at a company once where each bug had a really interesting field: root cause

I wish I could remember the values you could fill in, they were very intelligently chosen.

What I learned: if you didn't know what the root cause was, you probably didn't fix anything.

1000% this.

I've been in many orgs where root cause was either completely missing or completely missed the point. Recently I quit a company that thought they were doing RCA. The fact they sent out an email that there was an outage and then N number of hours or days later another email to say it was "fixed" and that they scaled or thought they found the problem and don't worry about it anymore. Literally we had weeks where the exact same outage occurred multiple times. And with every outage the exact same response.

So... I asked simple questions of leadership as to why RCA process was not implemented. Why RCA did not require a standardized template to be filled out as part of a production outage. Why a "5 Whys" approach wasn't being considered to truly expose the actual root cause. Why there wasn't any accountability.

At the end of the day failure cause is not root cause and many struggle conflating the two. Honestly when an organization doesn't hold true RCA as a critical part of engineering process I, personally, feel as though that organization will innevitably hit a glass ceiling. Among other problem areas the disconnect with RCA, for me, was why I couldn't stay at that company anymore. It was embarrassing watching from the inside as the same mistakes were made over and over with nobody the wiser.

So how did the leadership respond to your questions? Did they agree, and did things change?

They "took it as feedback". The problem was the leadership was convinced they had a real RCA process in place. They didn't.

There is no root cause.

https://www.kitchensoap.com/2012/02/10/each-necessary-but-on...

https://blog.acolyer.org/2016/02/10/how-complex-systems-fail...

http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...

The OP is talking about bugs not complex system failures. These are not the same thing. It might take multiple bugs to cause a complex system to fail, because complex systems often have enormous amounts of complexity built in, but a single bug is often a fairly unidimensional thing. You can identify the root cause of a bug.

Another way to look at that is that a root cause can be a set.

I've had more than one case at work where it would come down to bad requirements. Both systems worked exactly as specified and were bug free (for the issue in scope). They just had a different understanding of reality by design. Root cause here is some mixture of poor understanding of the problem domain by various staff.

> Both systems worked exactly as specified and were bug free

If the system is working as designed, then there really is no bug. A bug is a malfunction, after all.

I've always called broken systems that are working as designed BAD: Broken As Designed.

I'd agree. The defect (not sure you can call it a bug?) was in the design/spec not the system.

Was it like an list of predefined values? Where I work they do root cause analysis for everything, but with freeform answers so what you describe might be different from what I'm used to.

In general, I'm so used to RCA and layered mitigations (what one of our greybeards calls "belt and suspenders") that I don't know how quality happens without it. I'm a convert to the idea that if you can't fix a problem directly, the fix has to isolate or be as close to the problem as possible. Otherwise the bad state just ripples outward as complexity.

Unfortunately this was probably 20 years ago. I know it was a list of predefined values in a drop-down, but I'm not sure if there was a other/write-in field.

The gist was that the causes were appropriate and educational. Folks couldn't choose "user is an idiot", instead having to choose "the interface was confusing".

Well "the interface was confusing" doesn't really rule out "user is an idiot", but most likely will make matters worse.

I like having a set of broad predefined values (it helps with standardization, which helps with searching in the future).

But a freeform report is also necessary. How else are you going to adequately explain what, where, why, etc., the root cause was?

> if you didn't know what the root cause was, you probably didn't fix anything.

Yes! If you don't know root cause, then you don't know what went wrong. Not only do you not know what to fix, any shotgun debugging is likely to have only fixed one symptom, leaving the actual malfunction in place.

Almost all of the companies I've owned or worked for have recognized this with a simple rule: if you haven't found (and proven) root cause, then the bug cannot be closed as fixed. Any company that doesn't have a variation of this going on is a company whose products you can't trust (and a company I would prefer not to work for).

Saucelabs is very bad this way. We have tunnels flake out once in a while (I'm still convinced there's a concurrency bug in their tunnel implementation, based on missing events I've seen in test logs), but sometimes Sauce is just having issues.

When I'm seeing 100% failure rate, there's often nothing on their status page. Or there's some bullshit metric like VM acquisition times are double normal for, say, some Windows VM. But I'm not seeing 8% failure rate. I'm not seeing an extra 30 seconds. I'm seeing 100% failure rate, with long timeouts, and retries.

> if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999.

(The answer is 99.99)

It's probably not in practice, since that assumes failures are perfectly independent. If they are perfectly correlated, the answer is still 99.999. For most real cases, it will be between those extremes.

I think it depends on how you define availability.

1. Suppose we define availability as "at least one is up". If the failures are completely independent, then the probability of any one being down is 10^-5 (five nines) and the probability of all 10 being down at the same time is (10^-5)^10 = 10^-50 (fifty nines).

2. If we instead define availability as "all 10 are up" (which is essentially equivalent to one failure causes a cascading failure) then in the same scenario where failures are independent, this is (1-10^-5)^10 ~= 99.99% (four nines).

Although I agree with your assessment that the total uptime of the system is the multiplication of the individual systems, that doesn’t appear to be the point of the article.

> The problem is that they weren't monitoring from the customer's perspective. Had they done that, it would have been clear that oodles of requests from some subset of customers were failing. They would have also realized that certain customers had all of their requests failing.

This is saying that if you are small, all your failing requests are within the 0.001% that the provider is allowed to fail.

I suppose this depends on what how 99.999 uptime is defined in the SLA.

> if 10 of your cloud services each have a reliability of 99.999, all together the reliability is not 99.999.

It's like an episode of Dirk Niblick: https://www.youtube.com/watch?v=bCoGMYV3UPk

Reply on news.ycombinator.com