Hacker News Reader

The difference between a distributed system and a monolith is that in a monolith you have some idea what your single point of failure is, but in a distributed system you don't know your dozen single points of failure until they happen in production.

View on HN

Example: in 2020, a 6-minute hiccup in a network switch caused asymmetric connectivity, breaking their distributed leader election algorithm, making that system read-only, breaking a different system's heartbeat mechanism causing that system to assume a failure, enact a recovery and degrade in performance, causing 25% of API requests to fail for 6 hours.

https://blog.cloudflare.com/a-byzantine-failure-in-the-real-...

I love the dependency cycles that prevent things from being started up again from scratch.

Google had a secret storage system back in the day, that required special encryption keys to be entered when it got re-initialised. The keys were in a safe. Guess where the combination to the safe was stored? ;)

Reply on news.ycombinator.com