Comment by time0ut | Hacker News Reader

Comment by time0ut | Hacker News Reader

Parent Full thread

My company's legacy system (which still does most revenue producing work) has deployment problems like this. The deployment is fully automated, but if it fails on a server it fails silently.

I rarely work on this system, but had to make an emergency change last summer. We deployed the change at around 10 pm. A number of our tests failed in a really strange way. It took several hours to determine that one of the 48 servers had the old version still. It's disk was full, so the new version rollout failed. The deployment pipeline happily reported all was well.

We got lucky in that our tests happened to land on the affected server. The results of this making past the validation process would be catastrophic. Not as catastrophic as this case I hope, but it'd be bad.

We made a couple human process changes, like telling the sysadmins to not ignore full disk warnings anymore (sigh). We also fixed the rollout script toactually report failures, but I don't actually trust it still.

Ignoring disk full condition ! really.

Handling an out of space condition should be part of your test suite - it certainly was back when I looked after a Map reduce based Billing system at BT and that was back in the day when a cluster of 17 systems was a really big thing.

IMO that's too late of a condition check. You should keep a margin of free space for OS/Background applications just so you have enough life left in the application to detect and log it.

In this case, the application itself does not use the disk in question, so the old version kept chugging along obliviously.

We actually did have monitoring on the capacity of the disk in question. We discovered during the analysis that it had been alerting the responsible team all day. They had just been ignoring it.

I think the parent did well openly and honestly raising a personal example where missing a "basic" check caused near career changing problems - I applaud them for sharing a difficult situation.

I was concerned that it's possible to read your comment as if it was critical of the parent - was that your intention?

To clarify a little, I was neither responsible for monitoring the underlying hardware or the deployment systems in this case. I also didn't have access to fix it myself. It took me a couple hours to go from "random weird test results" to "full disk broke the deploy".

It was more a criticism of the OP's Sysad's -

Though for the system I mentioned one of the reasons they employed me as a developer was that I was also a sysad on PR1MES - an early example of devops maybe.

Reply on news.ycombinator.com