* Passwords hashed with SHA-512 _unsalted_
* No lockout / notification after hundreds of thousands of failed logins
* XSS vulnerability
* Able to change the configuration to upload executable scripts
* SSH password authentication enabled
And your top concern was that Slicehost didn't immediately shut down a machine? Would it have really slowed the hackers down for more than an hour anyway?
The ASF infra group did almost everything right but they still got burnt by esoteric details. The sshd conf had PasswordAuthentication set to no, but enabling PAM auth caused regular password auth to work. Did you know about that caveat?
And your top concern was that Slicehost didn't immediately shut down a machine?
tptacek never said that. He said it was a little disturbing that Slicehost took two days to turn off an instance that was exploiting a previously-unknown vulnerability in JIRA.
From the incident report https://blogs.apache.org/infra/entry/apache_org_04_09_2010 :
We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly. ( http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an... )
Apache's infrastructure group got their shit together quickly. The hacked Slicehost instance wasn't a threat to them anymore. Slicehost's inaction helped get Atlassian hacked, which I think is a bit disturbing.
Would it have really slowed the hackers down for more than an hour anyway?
Oh, you're right. Let's just leave hacked systems running. Shutting them down will only slow down hackers for an hour.
Shutting down a server doesn't stop the hackers using one of the other thousands of servers they have compromised. In this regard, trying to respond to an attack by shutting down the server is essentially a waste of time: the only winning move is not to play.
There were multiple mistakes made (and I'm not trying to cast blame), and I think we should be aware of the repeatable lessons for software development: hashes need to be salted, XSS matters, password lockout matters etc. If there's a lesson from Slicehost, it's that you shouldn't rely on someone else bailing you out.
I'm unclear why I'm supposed to care about Atlassian. I don't use Atlassian products.
I think there are lessons we should all learn / double check, but I don't understand why you chose to highlight this issue.
No, that doesn't mean hosting providers can ignore breakins on their own hardware.
Seriously, dude: breaks aren't like unfortunate Internet weather incidents. They're crimes. An actual crime took place using Rackspace computers. How do we even know what kind of forensics could have been conducted if Slicehost can't even acknowledge when their machines are compromised? Hosting providers have special responsibilities.
I simply expected more from you: as dfranke said, this is your opportunity to hammer home the importance of good security practice, and instead you're talking about something basically beyond our control (Slicehost's incident response policy) rather than the root cause (a large number of fairly basic security mistakes)
Please, publish a blog post series on each of the mistakes made, explaining each one and how not to fall into the trap. Explain why slicehost taking 48 hours to notify a customer before shutting down their machine is a big problem if it belongs in the list. I'll upvote every post once it hits HN.
The 5th point, as explained in the post... We actually tried to disable Password based authentication -- It is the ASF's standard policy for it to be disabled! -- we just failed at testing the configuration on Brutus. On our other machines, we had a block of configuration options we appended to the END of the sshd configuration, and these options successfully turned off password authentication, requiring ssh keys only.
On brutus however, we were missing a "UsePAM no", which meant pam went in and fell back to password authentication. This was a mistake, and we didn't realize until after the vulnerability when we were testing the machine, that its sshd was misconfigured.
Anyway, when is Apache going to get serious about a bug tracking project? :-)
Not familiar with JIRA but... couldn't you prevent this by either changing the ownership of the uploaded files or configure the web server not to execute any files in uploaded files directory.
But if someone phones you up and says we were hacked using one of your slices you would hope they would at least suspend access and investigate.
In computer security an hour can be a long time.
Were I a server provider I would take such notifications very seriously; a trivial investigation would surely have uncovered the illegal activity and shut them down.
(we are seeing more and more use of services such as slicehost and EC2 to perform attacks. I doubt that is a tag those companies want to carry!)
who said anything about owing it to them? :-)
Obviously Atlassian knows this also, because they didn't even bother putting an IP blocking rule into their firewalls. If Atlassian didn't even think that worthwhile, pointing the finger at Slicehost is dubious.
I hope Slicehost was simply going through their own process; notifying the customer and helping them to resolve the issue: the server may have been running a business that is multiple peoples' livelihoods. Those processes take time, and they have to balance the damage you're doing to the 'target' victim vs the damage to the 'compromised host' victim. Given that shutting down the compromised host wouldn't save the target anyway, I would also have prioritized my own customer.
If it was running a clients services they are compromised too and if that is the case, and Slicehost leaves it compromised, that is a serious liability problem.
There is a reasonably high likelihood it was a slice set up specifically to launch attacks (though the blog post does say compromised, so perhaps they have more info on the slice in question than has been made public so far). In which case leaving it running is, again, unnecessary liability.
I think tptacek and I are just surprised this didn't concern Slicehost as much as it, probably, should :)
My understanding is that Slicehost is responsible for the VM 'hardware', and you're responsible for what goes on inside it. There are exceptions for illegal behaviour, but I'm sure those involve checks and balances and processes, probably for legal reasons as well as for simple reasons of fairness.
We're not talking about instantly recognizable traffic here; this was likely HTTPS traffic.
It's trivial enough to spot if you review the logs and the information available. Anyone with an ounce of security knowledge would spot it very quickly.
Should the hosting company have circumvented your access controls and looked into your VM?
They do this anyway; as root on the dedicated machine they have access to your VM anyway :)
There are exceptions for illegal behaviour
This is illegal behaviour isn't it :)