Postmortem: Server compromised due to publicly accessible Redis
kevinchen.co
kevinchen.co
User-agent: ia_archiver
Disallow: /
Granted, I didn't read the original blog post. Were they distributing tools to make it easy? Or were they explaining how it worked?
In any case, if he hadn't written the blog post, someone else would have.
But really this is the OP's problem: he ran a system open to the internet without any apparent analysis of attack vectors. Server was compromised due to incompetence - not developer incompetence, but operations incompetence, a different discipline.
Probably not. The attacker would have gotten access to the machine. As there is a lot of software installed locally some probably have holes, which allows for full root access.
So I don't really believe it makes much difference for serious attacker if the attacker gets root access or normal access.
> "From time to time I get security reports about Redis. It’s good to get reports, but it’s odd ... in a software which is designed (as we explain in our security page here http://redis.io/topics/security) to be totally insecure if exposed to the outside world."
Regardless of the readme, the Lessons Learned should include both "know the security posture of the services you run" and "don't run anything publicly facing unless you are forced to, and only then if hardened well".
Everyone is fallible, things tend to go wrong. Insecure defaults in software hurt even expert devs (which this guy is if you look at his blog).
If your software is "designed to be totally insecure if exposed to the outside world" you should not ship it with defaults exposing it to the outside world. IOW you should not design software to be totally insecure by default.
In a well-prepared system, the administrator would:
#1 block input traffic on all the ports, except the required ones. as routine, after uninstalling any service, open ports are reviewed; even if a service is forgotten running, it won't be able to communicate.
#2 not run redis or any service as root user.
Not doing so is, to say the least, reckless, and with this assumption/context, it doesn't make much sense to theorize about software design.
But I am puzzled: a running process like redis is supposed to run in a chroot, and /tmp is supposed to always be mounted with no exec, the chroot and process should have been runned with seteuid/guid redis...
I normally have backend technologies confined in a jails with pf authorizing only inbound & outbound traffic on the legitimate interface+port, and you need no ssh access in a jail.
Looks to me there is a cascade of sysadmins wrong practice.
Devops and modern "full stack devs" seems to have lost their basics.
Perhaps this is what happens when you take a "dev" with no real sysadmin experience and make him/her responsible for these machines? It's like giving my neighbor's kid with a small dirt bike the key to my Harley and sending him out on the highway -- bad things are bound to happen.
It reminds me of the common solution that many "amateur sysadmins" have when they encounter SELinux issues: turning it off. :/
process poorly packaged crash and nothing in the logs.
You disable it, see there was a warning about a poor permission and shout at selinux for making you lose days worth of works by poorly logging.
Btw, I am a dev. But I have to sometimes go in production see servers because I need to help production sysadmins troubleshoot the weirdness of todays mis-performing easy to use solutions (selinux, systemd, rsyslogd, packaging).
When you have to recompile centos or debian packages to have them support vanilla features and you are a dev, it is boring.
Since none of these memory store solutions (memcached, redis, etc.) should be able to be connected to on public addresses it would be a nice feature to only allow them to listen on private internal IP space or sockets by default and fail to run if anything else is used.
This would help reduce everyone's attack surface by default and help prevent misconfiguration by the user or 3rd party integration software.
Um, no. You should absolutely be doing both. Defense in depth is the only way to remain secure in failure of any other component. A firewall WILL NOT PROTECT YOU FROM THAT.
For example, you just put on the firewall and leave a daemon running on localhost as root. The firewall allows connections from localhost by default (as most do). An attacker finds a hole in your webserver and is able to run scripts in the httpd context. The script makes a connection to the daemon on localhost, exploits it, and now has root privileges. With root the attacker modifies the firewall rules to allow access. You are now fully exploited.
Had your daemon been secured they would have only been a takeover of httpd and daemon accounts and not a full system compromise. Privilege separation is extremely important.
Of course one uses the system package manager when possible, and a specialized package manager like npm in most other cases, but not all is lost when one needs a recent version of some autotools monstrosity. In that case just install it using GNU stow. Then uninstalling is just an invocation of "stow -D" (plus deleting a single directory).
But that only makes sense for a box one physically possesses. Why on earth would one reuse a VPS? Nevertheless, I'm glad that TFA did, because the investigation was very interesting.
No, no, no, never! You can poke around the server to figure out how you got attacked, but afterwards the only way to recover is to rebuild from scratch. Someone had root on the system. For all you know, you're currently running a recompiled version of kernel/init, which will recreate all the original scripts when you're not looking.
> Lessons Learned
I'd also add: learn about Apparmor / Selinux and use them in production. Even if redis was running as root, you could still prevent most of the attack (no file writing, no execution, etc.)
This isn't very difficult, just make it bind to localhost by default.
In the last 20 years I've been working on systems, that is the default on way too much. It's only been in very recent history that things come more secured by default. In the past popularity was strongly correlated with ability to set up. 'Works by default with no security' is really easy to setup and reduces the number of complaints that "product is too hard to use, when with $x product instead".
I remember when I first learned web development and I absolutely had a belief that the defaults would protect me from royally fucking up. Postgres security defaults for example. And although this is obviously not true in reality, we can't let ourselves overlook it just because.
java 1114 elasticsearch IPv6 TCP localhost:9200
nginx 1169 www-data IPv6 TCP *:80
nginx 1169 www-data IPv6 TCP *:443
dhclient 1620 root IPv4 UDP *:68
sshd 1697 root IPv6 TCP *:22
Either way, you should always check the open ports and listening services in a given machine. In this case for a quick check you could use: https://www.isredisallsafe.com/
Default system wide failure is a redis problem. Any modern daemon should come with configuration files that a) don't allow the application to run as root unless uncommented by user action, and b) bind to localhost. Further configuration should be necessary to break the system as thoroughly as this user did.
We're professionals, we don't need to RTFM.