Nines are not enough: meaningful metrics for clouds
blog.acolyer.org
blog.acolyer.org
I feel like sometimes you don't know the bad outcomes, until they happen. E.g., years ago, my team had a fairly big outage/issue caused by S3 reads taking multiple of 60s. (That is, our latency graph had very large spikes right over 60s, 120s, 180s¹. These were reads in the typically in the 10 KiB to 100 KiB range, sometimes as large as single-digit megs — i.e., they should take milliseconds, maybe a second, not 3 minutes. It took a significant amount of back-and-forth² as it either only effected our bucket, or just nobody else noticed. But it slowed our ability to process down that we built up an incredibly backlogs. (Processing a file had previously taken <1s, and was now taking over three minutes in some cases, a 200x slowdown!)
This is still not covered by the S3 SLA.
Also had a different cloud provider where the "Create VM" API returned 200 OK. The VM never finished booting. The SLA was over the 200 OK, not the actual completion of the task. Basically, exactly the example in the article, with a real world "we're paying for this?" provider.
¹I'm simplifying. It was weirdly actually 70s, 130s, and 190s. I can only presume that's a 10s timeout and n 60s timeouts, somewhere.
²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?"
They can probably see their own latencies at a service level, but if this was only affecting a few tenants, their internal tooling might have security-related restrictions on "dig into the request characteristics of this tenant".
Some providers deal with this by making CSRs log a reason for each access that you need to be prepared to justify when audited, others just lock it down completely outside of particular request patterns (e.g. "provide a request ID"). Pros and cons on both sides really.
Response latency is often only sampled, so if you observe rare outliers, support might not be any visibility by default.
They may see that there's a spike of high-latency requests for the system, as a whole, on their chart, but need a customer's explicit permission & particular request IDs to run a "Select from request_log where ..." query.
Generally speaking, just because you see a spike on a graph doesn't necessarily give you leave to start digging through random customer queries.
>²Woe unto the person who doesn't come to support with request IDs. My impression the whole time is "these guys can't see their own response latency?"
... so I'm going to do the same. :)
It's also possible that while that latency is weird for your particular usage, it's totally expected for someone else's. That can make it very hard to find your problematic requests amidst the noise of everyone else's average day at the bitbucket.
the solution according to support is to set your own aws client config that kills connections if the timeout is larger than expected.
the 60s is defined in the default client config
I had to spend 6 hours analyzing data to figure out what factors of a query actually impacted response time. It required me re-learning advanced spreadsheet skills to find correlations in log data. We're now in the process of rewriting the agreement because this analysis was not done at contract time.
This is a topic not really written about that much.
Another favorite of mine is when the SLIs are written in terms of average rate ignoring spikes. eg, "100ms latency at 100 requests/sec". So the customer slams you with 6000 requests as fast as possible, sleeps 60 seconds, and repeats. That averages out to 100 TPS, so how come you're not hitting your latency SLA?!
[1] not 100% if that's what they are.
Embarassing recent downtimes for Google and AWS.
Seriously, some systems only need to be up when people are actually using them. It doesn't matter if they don't work out of hours or over weekends.
OTOH "out of hours" or "over weekends" is a very good time to make batch processes happen, so for some business services it might be better to not be up during hours, but reliably be up outside of them.
An other issue with that is when the service is used internationally / globally, or even just by 24/7 businesses, and even "over weekend" becomes not necessarily a thing.
This also often happens prematurely. If the team in the US needs live data entered in Asia then there's not much you can do, they system has to be global. But an often overlooked option (today) is running multiple instances of the same software with each only needing nine fives in it's region. Even if you do need live data it might be better to have another process shuffling it between international instances. Also helps with latency.
https://www.stanq.com/OpenVMS%20and%20High%20Availability.pd...
They can get an updated version from this company:
Even better, the hardware it runs on might be cheap on eBay. :)
For the most part companies have decided that high uptime guarantees aren't worth the cost. Or they start rolling their own systems to deal with it.
Unless you can force people to use your crap or your competition is barely working, it's not really acceptable. A typical customer has pretty reliable internet connection during normal operations and only rare long outages. Meaning that any unavailability and unresponsiveness that lasts more than a few seconds will be pretty visible and annoying to customers. And assuming your competition can do three-four nines, you'd need like four-five nines to do better, which is actually five-six nines as a target.
Edited to add: Wonderfully, when trying to post this I got "Sorry, we're not able to serve your requests this quickly."
What makes good sense in leasing unused factory capacity would make good sense in leasing cloud resources, eh?