Backups are so simple, and yet the only times people seem to realize their true value is when they don't have any.
At the very least, if you have backups, you might have the ability to rebuild/redeploy, but for any non-trivial system, without a comprehensive DR plan that is drilled regularly, even having backups won't save the business if the tooling to get those backups into a production ready state.
It gets more interesting when you're trying to backup a distributed database. When your durability strategy is to spread your risk across a lot of nodes in a cluster, and you're federating your cluster across multiple sites, traditional backups may not be a practical option. However, as I write this, I'm realizing that there's nothing simple about distributed anything, and saying "it's hard to..." may not be all that interesting.
Keep your backups on-site but off-line - you already have many locations. You should practice a location (and it's backups) going offline at the same time as a code bug screws with data in the active system.
We restore our MySQL and Cassandra data every day in our production DC from backups in the DR DC. It's the only way to ensure things are working. It's about 100TB of data, but things zip along nicely now that 100gig networks are (more) common.
Disagree. A high-confidence backup plan that includes continuous restoration is of middling difficulty to design. Many get backups wrong and only find out when their archives turn out to be incomplete.
If a Cloud/SaaS provider is going to claim to be your hassle-free one-stop shop, shouldn't they provide, as a matter of course, a feature to dump your data daily/weekly/monthly to a designated off-cloud site of your choosing? But they don't, because they want to pretend that they are impenetrable and 100% available.
The goal to hopefully influence companies into providing such a fundamental ability.
(Which works reasonably well as long as the damages are quantifiable and the other company is actually able to pay them....)