Hacker News Reader

Comment by mhandley | Hacker News Reader

Maybe it would be a good idea for certificates to expire slowly and randomly over 24 or 48 hours. In other words, if the cert has an expiry date of 12:00 UTC, Dec 6th 2018, then start to randomly fail connections at that time with low probability. The probability increases progressively during the next 24 hours until 100% of connections fail at 12:00 UTC, Dec 7th 2018. It's not like the cert is 100% trustworthy one minute and 100% untrustworthy the next minute. Having the failure rate ramp up slowly would give advance warning before everything has gone completely pear-shaped.

In the case of Ericsson, this might have allowed an emergency certificate update before all the O2 systems could no longer be automatically updated. Once your network is completely down, bringing it back up remotely is hard.

View on HN

No, this is not a good idea. Failures which are random are harder to diagnose. Something which works or not is much quicker to track down.

The simple fact is that this sort of problem should've been dealt wiht a lot sooner, and a failure to do so is sheer incompetence.

Where that incompetence lies, is up for debate. But given how these things usually run, it lies some levels above the people who didn't have the time to properly follow a poorly-described process in the face of demands to do other stuff instead.

Of course certificates should be managed properly, so this sort of thing should never happen. My suggestion is not instead of that, but in addition. The question is when your certificate management process has failed, what then? It's defense in depth.

In this case the certs expiry seems to have rendered it hard or impossible to remotely resurrect the systems. That's a very brutal failure when your whole network goes down in one go. Having random failures, with alarms going off before everything is down, is definitely better that having alarms going off with the entire network already down.

That can be achieved by having multiple carts issued with slightly different expiries. Without weakening enforcement of all other certs (incl. those belonging to companies that manage them properly).

This is immediately results in the “it took us more than 24 hours to work out everything was failing, it needs to be 72 hours”. The best thing about that is that it results in a self extending policy - the longer it takes to age out an expired cert, the longer it takes for the failure volume to become noticeable, and then the longer to work out that the problem is cert expiry, and so the more likely they’ll need an extension (again).

The correct solution for this problem is for people to correctly manage their certs, which they should be doing anyway because the private keys are sufficiently important you should know exactly which are in use and where they are in use.

Maybe it would be a good idea for people to put an expiration date check in whatever they use to monitor the rest of the machines. Or put the expiry date on a shared calendar.

We do both, and they start alerting two months in advance. This isn't rocket surgery, new, or much different than remembering people's birthdays.

But a surprising number of technical people are simply incapable of managing events farther out than next week, and a surprising number of their managers are similarly incapable of making sure someone treats it as important.

Makes me almost want to start a betting pool on who will be the next HugeCo to down themselves this way.

Edit: And let me join the chorus in saying deliberately inducing random errors as a way to draw attention is a terrible idea.

> whatever they use to monitor the rest of the machines.

"Ah, are we supposed to do that too?"

I think this could also be a good idea for phasing out public APIs -- instead of just taking an API offline, start to fail requests early at low probability, ramping the probability up to 100% over the course of a month or so.

Adding delays to responses can also work too.

Years ago, during the tail end of the Age of XML, I had a conversation with one of the admins over at W3C.

To make your app work you're supposed to cache all schemas locally but an inordinate amount of people don't. Their apps will be slow because of that, but work anyway. So tons of traffic goes to w3.org. They can't remove those schemas because then nobody can use them the right way (download them 1x and keep the result). But how do you get people's attention?

I pointed out to him that a human downloading the schema isn't going to notice if the request takes .3 seconds or 3 seconds, but someone loading that schema 100x in a loop is sure going to be incentivized to figure out where the extra 5 minutes of processing time came from. He allowed that I might be onto something there but I never heard back about whether he tried it.

You want to kill off an API, assign it resources (hardware, bandwidth) that get progressively smaller, or add artificial waits that get progressively bigger. Depending on how horizontally scalable your app is (is it easier to add a sleep or to pull servers out of the cluster?).

How long would it take before developers started wrapping their API and CDN calls in rapid-firing loops because of this?

It already exists and has borrowed the name "resilience engineering" from the construction and engineering fields. Netflix has some interesting blog posts on how they deal with transient faults and resilience in general. Implementing concepts like circuit breakers.

Have a search for libraries in your favorite language, I'm sure something will already exist. I've personally used Polly in .NET.

https://github.com/App-vNext/Polly

Are you insane? That would be fun to diagnose.

While I can totally see why this would be annoying to diagnose, throwing a specific form of a certificate expired error code could mitigate this issue.

At this point I wouldn’t even contemplate using X.509 without a fully automated PKI issuing ephemeral certs. Paging humans is not the answer.

Certificate monitoring would be a good idea.

Reply on news.ycombinator.com