statd 7680 690 0.0 743976 58492 ? Sl Nov26 20914:53 ./yam -c 1 -M stratum+tcp://binyu.crypto%40gmail.com:x@xmr.pool.minergate.com:45660/xmr
This was running under the statd user.
What do we do? We checked firewall, SSH, all seem OK. How do we go about investigating this breach? Do help!
I'm not very familiar with the tech behind BTC mining so there may be some obvious reason that isn't feasible. I was always surprised some evil company like EA hasn't added it to wait screens when matchmaking for various games. They could install whatever they want, target machines likely have solid GPUs, and they are sitting around waiting.
I also wonder about the legal aspects of this. Would someone need to opt in this? Is the energy cost of something like this legally distinguishable from sites that say... load a bazillion tracking tags and eat up your data cap?
Anyway, it made the average computer sound like a jet engine. That, and the fact that the average computer is nearly useless compared to a dedicated mining rig, makes browser-based mining probably not worth it.
It's pointed at xmr.pool.minergate.com:45660/xmr
XMR is Monero. It has a lower hashrate so i see how the attacker can make something out of this.
Are you sure it's not an inside job? Cause anyone with access to run this under statd basically owns you right now
Strange thing if we run 'top' from the main host, all containers running redis say 'statd' as their user; inside the container the user showed 'redis'. We removed nfs and all related files, and now it shows a user ID number. Is this something we should worry about?
Hopefully you're taking regular VM snapshots so you've got some logs they can't delete. Otherwise good luck, someone Bitcoin mining is probably clever enough to cover their tracks.
Realistically an breach bad enough that they have server control is probably through the web. The most common way I've seen is through various CMS code execution exploits. If your web apps allow file upload that's a really common way to get code running on the server as well
Would you how we could hire professionals to investigate this for us? And report it to appropriate groups..?
PS: These are dedicated servers :-/
Basic computer forensics needs a copy of the drive as unaltered as possible so you should start with that before running or installing anything. Basically don't run the server if you want to be able to get anything our of it.
If it's not a user data breach its par the course to reintsall and sweep it under the rug lol.
Next time make sure the server takes snapshots and dumps logs to an external place where they can't be deleted.
Have you ruled out an internal source that decided to use some of your "spare" capacity? Or a previously internal source that might not have had all their privileges revoked?
If your (root) password is weak then I'd not be surprised if that was the source of the infection.
You might see logins via "last", or via the system logs.
Is ssh only allowed by public key? (in /etc/ssh/sshd_config => PasswordAuthentication no)
Is Apache or NGINX running on the server?
Is PHP/Ruby/Node/Python running apps?
What ports are open in iptables (iptables -L)
What does /var/log/auth.log say?We searched the whole system for authorized_keys files and found one created in a /var/lib/redis/ of a staging container (with no firewall) on this host. We then came across the redis vulnerability https://kevinchen.co/blog/postmortem-server-compromised/ . A junior dev had spawned this container without help from dev-ops and hence left ports open.
What doesn't make sense to us is how this daemon (yam) was running under a statd username when the container doesn't have such a user, but the host does? Are LXC containers able to run daemons on the host?
This is because usernames don't exist, as far as the kernel's concerned. ps is resolving the process's UID to the corresponding name for the outside context, not the one inside the container.
I might be able to help you on investigating this issue. Contact details are on my profile.