Hacker News Reader

Hacker News

Comment by theDoug | Hacker News Reader

Full thread

theDoug

One line summary: “99.999% availability and strong consistency — without compromising latency”

(I work at G)

View on HN

5 minutes a year is really impressive.

Especially considering that all of AWS database solutions require a 20 minute maintenance window per week!

What are they doing during those 20 minutes?

Define "maintenance window" - the DB becomes read-only? Or unreachable? This sounds insane to me.

Not usually, but it sometimes will go offline for 5 or 10 minutes or more. It fails over essentially.

Actions that require database downtime/restarts can happen during the maintenance window. For example, if you change a DB parameter that requires a database restart, you can tell AWS to restart the DB during the next maintenance window. Same goes for DB engine upgrades and the like.

If you're running anything production-worthy, you'll be using one of AWS's multi-AZ/failover solutions for your production RDS instances. In that case, the database will perform a failover in those instances so there isn't downtime.

That depends on the parameter being changed; not all parameter changes are compatible with block level replication. Such as the MySQL InnoDB log file size, something that needs to be changed on every instance.

> In that case, the database will perform a failover in those instances so there isn't downtime.

Have you done that? I have and there's been downtime every time (~5m).

That doesn't mean they're unavailable for 20 minutes weekly, probably because of the nature of AWS's multi-AZ RDS setup.

At our shop, we haven't had any problems with RDS maintenance at all -- and we would indeed get paged for even 1 minute of database downtime, much less 20 (!). That said, we don't have that sort of monitoring in place for any of the test/staging DBs (which are not multi-AZ).

That's a window when they _can_ do maintenance, it's pretty rare for them to actually do it.

The SLA document [0] says 99.95% so perhaps you meant per month?

0 - https://aws.amazon.com/rds/sla/

You see the trick here, that's 99.95% outside of their regularly scheduled maintenance!

Oh! Now I see. I haven't used RDS for anything critical yet but thought it'd be a no-brainer.

insert mind expanding gif

In practice it works just fine. They just make you give them permission to blow your app up for twenty minutes a week (and wink and say maybe they won't actually do it this week).

With respect to SLA's the two important things in my opinion are:

* How precisely is it measured? * What happens if it is not met?

As observed by the other posters, scheduled maintenance doesn't count.

The RDS service is literally scripted EC2 instances running MySQL/Postgres/SQL Server etc. Anything you do in the RDS console/API runs (or schedules) scripts on EC2 instances. If you would have downtime on any maintenance of your self-managed DB instance (say, upgrading the DB engine), so will RDS.

Some people get disillusioned at this fact, thinking RDS was something more magical. But AWS does offer the Multi-AZ option, which does the maintenance on the standby, fails over, and then performs the same on the main instance. It's effectively transparent. RDS also makes backups, restores (including point in time), encryption using KMS etc. really easy. If you don't have a full-time DBA, RDS provides for a very easy to use DB with virtually no maintenance required, but Multi-AZ is absolutely required for any kind of production deployment.

AWS's answer to Cloud Spanner is much more likely to be a future cross-region replicating version of DynamoDB or Aurora, not RDS.

As far as Multi-AZ is concerned, it's a bit of a wash. A month ago when the east coast had its issues, Multi-AZ didn't do anything. Every time I need to reboot an instance, with failover it is down/inaccessible for the same amount of time as if I had just rebooted it plainly.

I'm sure there's scenarios where it's worked well for people, but I've never seen it happen.

I see multi region saves many disasters. Multi AZ seems a waste. What percent of failures only hit 1 AZ? Your loss is for sure (higher latency, bandwidth cost).... unclear what the real world gain is of multi AZ.

I’m in marketing and not here to pump our own tires, but Spanner is /the/ database beneath so much of Google. We trust and rely on it, so it’s very exciting to me to see other big companies who need mission-critical /and/ global scale performance do the same.

I'd love to use it, but I can't reasonably champion Google's services at work when I'm not allowed to use it for side projects.

When is it going to be available for non-companies in Ireland?

It's about VAT collection. If you sign up and warrant that you do it yourself, you could click the button. However, your note reminded me to kick off a specific email about Ireland as a possibility.

Disclosure: I work on Google Cloud, but I'm definitely not a lawyer, tax lawyer, accountant or any such financial type.

It is, but the details of the SLA aren't that great.

https://cloud.google.com/spanner/sla

"'Downtime' means, with respect to any Production-Grade Cloud Spanner Instance, more than a five percent Error Rate for the instance. Downtime is measured based on server side Error Rate. 'Downtime Period' means a period of five consecutive minutes of Downtime with a minimum of 60 requests per minute. Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods."

For monthly downtown between 99.0% and 99.999% the customer is credited with 10% of the monthly bill. The customer is required to track the uptime and request an SLA credit.

> Intermittent Downtime for a period of less than five minutes will not be counted towards any Downtime Periods.

That's unfortunate. The most frustrating part about cloud services are exactly these kinds intermittent performance and responsiveness issues.

It probably won't count network downtime either, since it's measured at the server, not the client. Even if your client is in GAE...

That's like saying 'the most frustrating part about cloud services is that they use the internet.'

You're right, but that's basically 'by-design' as far as the internet is concerned.

Part of the problem is that computing the downtime at a higher resolution, in a way that scales from low-traffic users (the quoted minimum of 60 requests per minute is also not a coincidence) to very large ones, is not trivial.

You have to consider the rate at which the raw data is collected from the replicas ---and the jitter across all of them!-- then the rate at which that data is aggregated, and so on. (The above assumes that Cloud Spanner uses Borgmon for monitoring.) Any hiccups in the collection itself will result in less noise if you look at 5m vs e.g. 1m. You could increase the sampling rate, but then you're making your DB replicas spend more precious cycles on work that is not serving queries or replicating data.

That really should be in the title of the article.

Reply on news.ycombinator.com