Hacker News Reader

Comment by bhouston | Hacker News Reader

I am always scared to put too much trust in new Object Storage services. I love Hetzner and similar but until this new service has been around for a while, I'd only use them for stuff I can afford to lose. From the outside as a consumer of these services you do not know how robust they actually are on the inside.

I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building:

https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-dam...

View on HN

"Cloud 3-2-1" to the rescue...

I take the typical formulation (e.g., [1]), and translate it into:

- Keep 3 copies of your data: production + 2.

- Keep 2 snapshots, stored separately.

- Keep 1 snapshot on-hand (literally in your possession), or with a different provider.

It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains.

[1] - https://www.backblaze.com/blog/the-3-2-1-backup-strategy/

I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested.

However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it.

[0] https://www.tigrisdata.com

[1] https://www.tigrisdata.com/docs/migration/#starting-the-migr...

Backblaze is 3x less expensive and offers more generous free allowances.

> Access small objects at close to Redis speed

This is a massive claim that unless they're storing everything in memory I don't know how you can get close to Redis speed.

Even the flash-based Redis alternatives can't get close (but are good enough)

Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery.

From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.

Maybe if OVH had installed an automatic fire extinguishing system or an electric mains cut-off switch, the firefighters wouldn't have struggled with meter-long electrical arcs and could put out the fire quicker.

Yes, that hosting center was unsafe as hell. All of OVH's early year featured a lot of duct-tape and daring corner-cutting - that is how they were incredibly cheap. But nowadays they are just as mainstream industrial as their competitors and the 2021 fire marked the end of an era... It is sad that it happened as they were already turning the company around to being an established player.

A great writeup of the incident and its context, three years afterwards: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...

> But nowadays they are just as mainstream industrial as their competitors

Shame their control panel is massively behind their competitors, it's the worst cloud control panel I've had the experience of using, even worse than Azure and Google.

It's slow slow slow slow slow and horribly buggy, did I mention it's slow?

People seem to have this misconception that "cloud computing" is some kind of magic bullet that guarantees 100% durability. News flash, your data is still being stored on physical hard drives, in some data center. If the building burns down in a fire, of course your data will be gone, the hard drives are hosed, it's not magic. You really have no one to be mad at other than yourself...

Most of the "big" cloud providers aren't as negligent as OVH though. That entire datacenter was just one blunder after the other (wood structures, barely functional fire suppression, no proper power shut down, insane replication strategy that caused most of the replicas to burn in the very same fire in the same building...)

> If the building burns down in a fire, of course your data will be gone

From https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu... :

> S3 Standard, [...] redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.

I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that.

another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.

Switch builds their DCs with two roofs...

https://www.switch.com/switch-shield/

The Switch one I'm in (Grand Rapids Pyramid), the servers are about 3/4's underground.

S3 durability is pretty much the only thing I consider worth using EC2 for.

S3 egress, on the other hand, is so expensive you can often justify putting a writethrough cache for your entire working set at Hetzner...

IIRC Cloudflare R2 is designed as a cache for S3 and has free egress

edit: apparently not writethrough: https://developers.cloudflare.com/r2/data-migration/sippy/

No, R2 is not an "S3 cache". It's a replacement for S3 with no egress fees.

Yeah, don't use Sippy. It just stopped working after 2 weeks on multiple buckets and support came back with "have you tried turning it off and on again". It's not production ready.

Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.

You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.

> It's not like there is a mirror datacenter just two blocks away

Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options)

It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region

[0] https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform

It literally was one building in france, that burned to the ground: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...

If us-east-1 ever suffered a “FULL” data loss, it would be a company-ending event for so many companies that it would practically end society as we know it.

OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.

Is the any GCP documentation on that? Sounds like a far bigger risk going with GCP than AWS if all zones in a region are in the same building.

It just caught those who rely on one provider, which is never a good practice.

rclone to somewhere else at least.

And 3-2-1 rule for backups if you're serious.

One of my VPS was lost in that fire, but since my backups where somewhere else, I didn't lose much.

The trick is not to put all eggs in the same basket and it's easier to do that when you're paying much less than you'd pay on providers like AWS.

I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.

Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.

As everyone knows, this is in sharp contrast with serious engineering operations like Amazon or Google, where there are no disagreements on technical designs. Ever.

Reply on news.ycombinator.com