Hacker News Reader

Comment by derFunk | Hacker News Reader

What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did. You probably didn't. So save your configs and set this machine up from scratch.

View on HN

I was gonna say much the same thing. Once a machine has been compromised, you've got to format and re-image it. The machine in the article could well still have a rootkit installed. Not to mention he doesn't seem to know the initial attack vector, so any security updates are meaningless if someones credentials have been stolen.

Finally, if you're really paranoid, you also have to flash the bios in the machine too as there are rootkits that can potentially survive formatting...

I'd add that you don't even need to have the machine actually be compromised.

If you think it might be compromised, if you have reason to believe that perhaps something bad happened, you probably should re-image it. There may be smoke without a fire, but it's not worth the risk, is it?

This seems like a great reason to virtualize everything possible.

True, but even this isn't always perfect. I was told recently that Xen shares the kernel memory space among VMs so if once is compromised at the kernel level all of them could be compromised. This person dislikes using Amazon for this reason because he can't be sure his box isn't popped through another VM.

Generally speaking, Xen-style virtualization is where each VM has their own copy of the kernel and their own memory space. There are new-ish features like transcendent memory which can blur the lines, but I don't think they are widely deployed.

Systems like docker, LXC, and virtuozzo are where you run into the issue you describe.

This is correct, of course, in principle, but in the real world its often not feasible. As a freelancer, the author likely spends much of his time working on small business blogs and brochure sites.

These are frequently set up on a single server, containing code, database, and any file storage necessary. They're frequently written by cloning and modifying a framework, making reinstalling from a known good copy difficult to impossible. They often have no version control and are frequently configured to allow the webserver to modify code, so there is rarely a known good copy of the site code. In short, what's running on the compromised server is often the only usable iteration of the site's code available.

I hope nobody on HN would willingly set up a site like this, but the fact remains that they exist and need to be maintained. Attempting to clean a system in place is nowhere near 100% effective, but it's much more effective than doing nothing, which is exactly what will happen if you try to tell a client that they have to be offline for days, lose several months of data, and pay someone to rewrite half their site from scratch because you want to format their server and attempt to rebuild it.

If moving the project to a new server is not feasible, then how does the client backup their data?

I refuse to work on projects that cannot be installed from scratch in a local VM.

Definitely. Also, I don't know anything about their relations, but if this is a client machine, first thing to do would be to contact the client telling him that his box is compromised and if he wants me to investigate. Otherwise, from a non-technical person view, the whole (awesome) story looks like somebody attempted to do something with e-mails and he failed.

At first I felt bad that I didn't know all the tricks the author knew, but then I realized I would never, ever try to clean a machine. I would copy (and verify) and config files I needed and just spin up a new vm. Its not worth the time and there's a slim chance you'll actually catch everything. Funny how he didn't just run rootkit hunter. I wonder if it would have detected this hack.

I'm also starting to think that the move from Linux being a sysadmin-only OS to being something that can casually be spun up by even the most jr of developers, means we're having a Microsoft circa 1997 moment where security needs to be commoditized. I wonder if any of the few commercial AV's for Linux are updated frequently enough to catch hacks like these.

I think he's too clever for his own good. This machine should have been considered beyond salvage, remove any precious data (you did have a backup, didn't you?) and re-image. I've had to recover data from hacked boxes a few times for 'brand new customers' and the first thing I do with a system like that is to make sure I get a console wired up and the uplink disconnected. No point in taking chances.

Save any data you need and consider the machine totally compromised.

By that, I mean you need to consider any IPMI processor compromised as well, since these can be attacked (or their secrets stolen). Basically rebuild the machine from as low a level as you can. Depending on the brand of server hardware, you may need physical access.

Until you restore from backup: then you probably have a compromised system again.

If you restore data from backup you're fine. Configuration should be re-applied. Unfortunately you need to check every file with an executable bit that comes out of your backups - review shell scripts, recompile executables.

It's rare that attackers tamper with data, usually they just leak it. Source code may be targeted though.

the risk is lower with data files, but it's not gone.

buffer overflows in non-executable files means it's easy for someone who wants to, to recompromise a machine: pdf, gif, jpg, whatever.

Reply on news.ycombinator.com