Hacker News Reader

Comment by michaelt | Hacker News Reader

In my experience, it's very easy to achieve high uptime through luck. If I only run a single server, and it has a 10% chance of failing in any given year, I have a 90% chance of achieving 100% uptime in a given year.

In my experience it's also very easy to think you've got all your bases covered when actually you haven't. I'm protected against mains power failures by a UPS and a generator - but a UPS switchgear fault can cut my power even without a mains power outage. My server has dual power supplies and network cards - but that won't help me if a clumsy worker sent to replace the server above mine unplugs mine by mistake. And so on.

If I think I'm doing a better job than a billion-dollar corporation with hundreds of thousands of servers, does that mean I am? Or is it more likely I'm fooling myself?

View on HN

I also have a personal dedicated server that never crashed or restarted this year. However I am not sure how much it was actually available. I know for a fact that there were multiple network issues at OVH. I also know that had my server been home, it would have been worse (Optimum residential is awful).

The server not failing is not the only outage mode.

Indeed. People do not realize just how advanced the reliability infrastructure of those services is. Things like diesel power generators have been baked into cloud datacenters for what, a decade now? Probably longer. Show me your alternative power source when the power goes out (and power does disappear, everywhere, eventually).

Diesel generators have been baked into my on prem equipment room for at least 40 years

For you average person working in an average office if the power goes out you're not going to be working anyway, so it doesn't matter if your server is offline too.

This so much.

And an excellent corollary to this is that when you have lucky 100% uptime there is no incentive to optimize mean time to recovery.

Sure the raspberry pi in your closet has been running fine for years, 100% uptime, but then a component fails. Do you have a replacement on hand? Are you continuously monitoring it to know it went down? The component failed at 3am, did it page you? Did you hop right out of bed to rush to fix it?

Single systems can have really nice uptime until they don’t. Then you are hoping that the people on hand can repair what’s going on after months or years of never having to do that. Mean time to recovery might be a week while you wait for new hardware or a few hours while you google some error message you’ve never encountered.

People can run their own systems if they want to, but they shouldn’t confuse good luck with rigorous engineering.

Do you have some external ping test running continuously? If not, you really have no idea.

Of course, how else would I know the exact failure times, my service monitoring only polls every few minutes. That doesn't help me with my knowing how my service is performing though, I'm not offering a ping service.

More importantly, I do know I had an error loading twitter at Thu 11 Nov 04:17:01 GMT 2021, however my websites (and google and hackernews) were working. At 17:53:01 GMT Twitter took over 2 seconds to load it's first http page, far beyond the normal 400ms. Google took 23 seconds to load this morning, BBC News just 0.071.

On the other hand I also need to provide services which can't cope with outages measured in milliseconds. Good luck with complaining to a cloud provider that your traffic vanished for 4 seconds. Those services thus have multiple connections on independent hardware and circuits with no single point of failure

Reply on news.ycombinator.com