Hacker News Reader

As a seasoned system administrator, I strongly advise you against unattended or automatic upgrades. In a Linux or FLOSS eco system components are very strongly coupled but loosely developed. Say glibc is base of nearly every server application but it is and server apps are not regres tested for each other. Which will or may lead to incompatibilities, down times or performance issues even on edge cases data loss. So please don't.

Your concerns are not an issue with a distro like Debian where the only upgrades are for backported security fixes (or, when an announced point release comes around, critical bug fixes). I worked for several years at a shop with dozens of Debian servers, and nightly unattended upgrades very rarely caused issues, and never caused anything truly serious.

> Your concerns are not an issue with a distro like Debian

Ten years ago, I had a Debian server completely wedge itself on an update. glibc was hosed, practically nothing would run, and I couldn't untangle the mess. Had it been unattended, downtime would have been even longer than the hours it took to rebuild the box from scratch.

This was not the first or last time such an event would occur, merely the most severe.

Updates break things. Anyone who claims otherwise is either incredibly lucky, or incredibly inexperienced.

But if you can't flip a switch to deploy an identical server and/or restore from a backup image, aren't you in deep shit anyway? Your failure plan is to rebuild the box from scratch?

For a one-off budget server in 2002-2003? Yes, yes it is.

For a highly-available mission-critical infrastructure in 2013, the failure plan is also to rebuild from scratch, because "failure" means redundant and backup systems have exploded. This is most likely to occur when you automatically roll out untested changes to your infrastructure.

In either case, you won't be sleeping tonight. Or possibly tomorrow night.

Debian is very stable. I recommend running "safe-upgrade" nightly -- and keeping an eye out for when you need a "dist-upgrade" (typically new kernel images -- and other stuff that should be tested (eg: test that the server actually boots -- not to mention do a reboot to use the new kernel).

I'm not sure how good the Ubuntu LTS relases are, but Debian has always been great at both keeping a stable system up, and keeping the upgrade to new version as painless as possible.

In the typical case, you only need dist-upgrade for upgrading to a new Debian release (which is every 2 years or so and is big enough news you'll probably hear about it). When there's a security update for the kernel, a regular upgrade/safe-upgrade[1] is sufficient to get it. Of course you must schedule a reboot yourself.

[1] "upgrade" if using apt-get, "safe-upgrade" if using aptitude

Ah -- I usually use "apt-get safe-upgrade" for scheduled upgrades, and aptitude for all manual work.

So I had the effect right, but the commands mixed up :-)

While you are absolutely correct, no one is regression testing the updates potentially coming into their project box/gameserver/etc, even if they perform the apt-get upgrade manually.

Might as well set it automatically and stay protected <shrugs>

I prefer to be actively administrating my server, so that I know if something does go awry. If I'm asleep, it may be a few hours before I realize that an update broke something. On top of that, I clone my servers and test upgrades in a development environment as much as possible, before allowing an update to go live. As long as you're on top of the updates, a few days between automatic and manual shouldn't have much effect.

Another really nifty way to achieve pretty much the same is to use iptables' built-in limit module like so:

  -A INPUT -p tcp -m tcp --dport 22 -m state --state RELATED,ESTABLISHED -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 22 -m state --state NEW -m limit --limit 3/min --limit-burst 4 -j ACCEPT
  -A INPUT -p tcp -m tcp --dport 22 -j DROP

Limit-bursts allows that many packages through before any limit counter starts, if any of the other packages afterwards also do not garner any response, a counter counts up for every further package, effectively blocking everything after the first 7 packages arriving within a minute. (You could also do 3/hour, for example, if you think you need it.)

The limit module is a beautiful thing, since you can also use it to limit log messages generated for events etc., using the same syntax, like so:

  -A INPUT -m limit --limit 1/hour --limit-burst 3 -j LOG --log-prefix "iptables denied: "

Awesome, now I can DoS your SSH with only 3 requests per minute.

You can DOS your own IP from being allowed in..... But not mine.

The `limit` module operates system-wide; one must use the `hashlimit` module to differentiate based on IP.

Both limit and spindritf's suggestion will mitigate the attack, but they won't notify you about it. In most circumstances, I'd prefer to skip on the thousands of notifications per day, however I sometimes like to know about every detail on a server, and Fail2ban gives me that level of control, without the need to tail a log file.

Security is a tradeoff. In this case, unattended upgrades lead to increased downtime.

(Why? Because you didn't read the caution note that upstream sent along to you, that's why. You just woke up to the alert saying that your system is down.)

If you have a reasonable environment, you should have alpha or qa servers that look just like production, except that they aren't visible to the outside. Always test changes there before upgrading on your production machines.

Security is a tradeoff. In this case, unattended upgrades lead to increased downtime.

Agreed. Indeed, the parent explicitly shows one possible trade-off:

Especially for your hobby projects or personal server because odds are that you won't always have the time to act on every security advisory.