top comment:
> If it was unnoticed for years then the sysadmins were shit. Why is this newsworthy?
>
> Any sysadmin worth their salt will be monitoring the systems they manage. This thing made outgoing connections every 15 minutes using a cron job. Any 15 year old can create "malware" like this. If you do not notice this for years, you shouldn't be allowed to manage servers.
> compromised via Joomla and Wordpress exploits
Lots of Wordpress installations are not run by someone who should know better.
Hell a lot of servers on the internet are run by people who shouldn't manage servers, not because they're lazy, but because they're aren't admins. Many tools have been created to let people without knowing how to do things do them but the security of those tools has not kept up.
I've apprenticed as a junior sysadmin in the past, and I can easily see how a professional operation would detect or prevent this by design. But a lot of those methods are more than a little over the top for a one-of, single-user, low-end VPS.
Ideally you would want a HIDS system for small deployments (i.e. a single server) but there doesn't seem to exist too many of those and there's little information about them online (as far as I can find). OSSEC is one example but I'm not convinced it installs too cleanly. At least FreeBSD has a package for it however. Actual configuration would be a mystery.
I think information about this sort of stuff (intrusion detection) seems to be hard to come by in general. At least stuff that is written by knowledgeable people. I haven't seen much about it from major vendors either - it seems to be a black box kind of technology.
I'd love to have a "lightweight" simple daemon that does this. I guess ideally you would monitor logs for stuff, user and group modifications, cron additions etc.
And then hope that your configuration prevents the attacker from getting root or if they do, then they/their automated scripts are not prepared to break them.
FreeBSD with a high enough securelevel allows you to set files append-only or immutable so that not even root can touch them.
But then updating the index becomes problematic, you would need to create delta files (which themselves are also immutable) that contain the changed information and combine the data.
Even then, I'm not convinced you could get that much valuable information from a full filesystem hash (or whatever) scan. The amount of information would probably be overwhelming. You would need to apply heuristics or filters. Checking filetypes with libmagic would work, mostly. You could only alert on new executables or whatever, etc..
I recommend it though
Also if you run anything like phpbb or wordpress, make sure you keep it updated.
I also like to disable Apache's suexec (as I once had a rootkit that exploited it, and I don't require it for my systems), limit inbound connections with iptables, and limit ssh access to certain ip addresses. Also, get rid of or disable access to any services you don't really need (like ftp, bind, etc).
It's also very important to check all your php/perl scripts for file/sql injection vulnerabilities, as that is one of the most common methods of infiltration.
Fail2ban isn't foolproof, incidentally. I once nearly caught a worm despite a limit of 6 login attempts per IP, because I had created an account with username and password both "test" that I had since forgotten about, well before I exposed that particular machine to the internet. Oops! An automated script managed to guess that in well under 6 attempts! Fortunately I was physically present at the time and noticed the abnormal load within a few minutes, and killed the offending process (doubtless a local password cracker for privilege escalation) before it could do any damage. Lesson learned there, and it also taught me the value of knowing your machine well enough to sniff when it's not behaving right. On a Windows machine I would have written off the abnormal load, as abnormal load is normal on Windows.