Hetzner object storage degraded for >1month

status.hetzner.com

35 points·Palmik··4 comments

If a backup falls in the forest and nobody is there to hear it, does it make a sound?

Backups are only as good as your last restore attempt.

Even AWS ran into this, when they accidentally zeroed out a service in one AZ and found that a combination of the dependency tree and timeouts had made it extremely challenging to spin a data center back up from zero. Instead of minutes it took hours and several false starts. It wasn’t stated but I suspect some of that has to do with the fact that if you’re spinning up a new AZ it contains no customers and no data, while spinning one back up needs to contend with retention and related event queues. Though they don’t actually spin up new AZs very often so what do I know.

This is the one thing I found crazy with the Erlang pattern that a lot of modern orchestration seems to be emulating in other languages.. If you put fixed limits everywhere on the theory that requests will just come back after you let them crash then you basically have everything having some hard to fathom upperbound that flips everyone from doing a long request to trying an infinite number of long requests..

The big rule with that is dependency loops. Every time someone creates a service that breaks up its own task list into subtasks and sends them directly or indirectly to itself instead of to a subordinate, there’s some traffic level where it hits the Dining Philosophers and deadlocks.

Recursion should never be used in distributed computing. Always modify it into delegation without recursion.

You don’t even need fixed limits to run into this. Resource exhaustion either in the queue or closures for started but suspended tasks will hit implicit limits eventually.

Absolute clown show. Do not trust these people with your business.

Reply on news.ycombinator.com