Hacker News Reader

Comment by fsniper | Hacker News Reader

As a seasoned system administrator, I strongly advise you against unattended or automatic upgrades. In a Linux or FLOSS eco system components are very strongly coupled but loosely developed. Say glibc is base of nearly every server application but it is and server apps are not regres tested for each other. Which will or may lead to incompatibilities, down times or performance issues even on edge cases data loss. So please don't.

View on HN

Your concerns are not an issue with a distro like Debian where the only upgrades are for backported security fixes (or, when an announced point release comes around, critical bug fixes). I worked for several years at a shop with dozens of Debian servers, and nightly unattended upgrades very rarely caused issues, and never caused anything truly serious.

> Your concerns are not an issue with a distro like Debian

Ten years ago, I had a Debian server completely wedge itself on an update. glibc was hosed, practically nothing would run, and I couldn't untangle the mess. Had it been unattended, downtime would have been even longer than the hours it took to rebuild the box from scratch.

This was not the first or last time such an event would occur, merely the most severe.

Updates break things. Anyone who claims otherwise is either incredibly lucky, or incredibly inexperienced.

But if you can't flip a switch to deploy an identical server and/or restore from a backup image, aren't you in deep shit anyway? Your failure plan is to rebuild the box from scratch?

For a one-off budget server in 2002-2003? Yes, yes it is.

For a highly-available mission-critical infrastructure in 2013, the failure plan is also to rebuild from scratch, because "failure" means redundant and backup systems have exploded. This is most likely to occur when you automatically roll out untested changes to your infrastructure.

In either case, you won't be sleeping tonight. Or possibly tomorrow night.

Debian is very stable. I recommend running "safe-upgrade" nightly -- and keeping an eye out for when you need a "dist-upgrade" (typically new kernel images -- and other stuff that should be tested (eg: test that the server actually boots -- not to mention do a reboot to use the new kernel).

I'm not sure how good the Ubuntu LTS relases are, but Debian has always been great at both keeping a stable system up, and keeping the upgrade to new version as painless as possible.

In the typical case, you only need dist-upgrade for upgrading to a new Debian release (which is every 2 years or so and is big enough news you'll probably hear about it). When there's a security update for the kernel, a regular upgrade/safe-upgrade[1] is sufficient to get it. Of course you must schedule a reboot yourself.

[1] "upgrade" if using apt-get, "safe-upgrade" if using aptitude

Ah -- I usually use "apt-get safe-upgrade" for scheduled upgrades, and aptitude for all manual work.

So I had the effect right, but the commands mixed up :-)

While you are absolutely correct, no one is regression testing the updates potentially coming into their project box/gameserver/etc, even if they perform the apt-get upgrade manually.

Might as well set it automatically and stay protected <shrugs>

I prefer to be actively administrating my server, so that I know if something does go awry. If I'm asleep, it may be a few hours before I realize that an update broke something. On top of that, I clone my servers and test upgrades in a development environment as much as possible, before allowing an update to go live. As long as you're on top of the updates, a few days between automatic and manual shouldn't have much effect.

Reply on news.ycombinator.com