How your data gets compromised in an IaaS cloud: Vendor tells all | Hacker News Reader

How your data gets compromised in an IaaS cloud: Vendor tells all

cloudsigma.com

26 pointscloudsigma24 comments

This is a big issue in certain verticals. In my early research for AR I looked into the interaction of HIPAA (an American privacy law for medical information) and cloud hosting. My brief educated layperson's conclusion: sensible default settings at your cloud service of choice almist certainly lead you to be OMGWTF noncompliant. I immediately moved medical providers out of scope, because it looked like there were, minimally, several months of engineer time needed to merit a finding of compliance, plus whatever costs/effort it would take to deal with the lawyers.

A lot of the compliance issues come from the mixing of infrastructure and the software and networking layers with many IaaS providers. In our cloud only the customer has root access and file system visibility. Essentially the cloud vendor then needs to demonstrate compliance with physical access and data protection/data leakage areas as employees don't have the ability to view data. This isn't a typical situation and for most clouds that are more like IaaS/PaaS hybrids it opens quite a can of worms.

As a Swiss based cloud currently we'd be excluded from many US industry sectors which required domestic hosting. This will change shortly (can't say more) and when it does we'll be working to put in place the necessary coverage/compliance certificates to expand into these sectors.

Best wishes,

Patrick

My layman's conclusion of HIPAA and cloud hosting was that the language of the law was vague and technologically out of date, and that depending on how you interpreted its requirements you'd be either OK or OMGWTF noncompliant.

It's too bad because the medical space is in dire need of innovation and there's a lot of money to be made.

There's definitely huge opportunities in the medical sphere. We do have customers from this sector in our cloud although how they handle client data is always very strictly controlled and I think its fair to say that only a subset of the potential is being realised properly in terms of the broader market.

Kind regards,

Patrick

I don't understand why a zero wipe isn't sufficient when provisioning the storage. At least for this purpose it would seem to achieve the same result as encryption with much less complexity and no ongoing overhead. AWS takes a long time to provision new EBS storage, does anyone know what's going on there?

Simply overwriting with zeros isn't a 100% fix by any means and actually it does involve writing perhaps 1TB of data in sequence. For any storage array that's a lot of data to write. It doesn't matter if its 0s or something else. Better is to use random data but even so you still have forensic techniques to revert this.

You say encryption is complicated but actually as a vendor its implicit in our system. As a customer you just mark the drive upon creation and then its invisible to both you as a customer and the cloud servers that are using the drive. That's really the whole idea, to make security measures that are convenient so people actually use them. Our customers see usually about a 10%-15% performance difference and we've got pretty high storage performance to begin with so it rarely means taking a performance hit compared with other platforms. We also allow multiple drives so users can categorise data by drives for encryption or not. Of course I'm biased regarding performance but the principles stand.

The other point of the blog post is to ask; what do other vendors do and do their customers have the ability to find out? Security through obscurity isn't an acceptable approach in the cloud. Everyone needs to be transparent and work to build confidence through solid information and education on how to use the cloud securely and effectively.

Kind regards,

Patrick

Better is to use random data but even so you still have forensic techniques to revert this.

How does one revert this without physical access to the drive?

We overwrite our EBS volumes with zeros before deleting them and I was under the impression that should protect us against leakage to other customers.

Naturally it can't protect us against a malicious amazon employee pulling the drive physically (or taking snapshots without our knowledge), but frankly I don't see how your "vendor encryption" helps with that either.

If all I do is set a checkbox to "mark the drive for encryption" then that means

   a) You have the encryption key, I don't.
   b) The checkbox could just as well be a placebo and not have any effect.

Thus we're back to square 1 and the same old question: "Do I trust you?"

No need to take a 10%-15% performance hit for that.

You raise a number of interesting points.

Firstly we can't comment on the arrangements of other companies for whom we don't have visibility. As a customer you can of course ask them and one would hope they are able to provide you with a full answer. On the blog we raise and answer (in our case) the various aspects for data storage, not just of security but also legal issues and data migration aspects. How many customers currently using an IaaS cloud can answer those questions or get their vendor to provide answers? Building confidence in cloud computing is all about transparency, education and creating secure ways of working. Different users will choose different solutions and regimes that they feel are 'secure' for them and we are all for that. We'd also like people to be able to make informed choices which means having the right information.

Secondly, there is a big difference between a vendor that has sole root access and full visibility of all your data and one that doesn't (as in our case); in our cloud the customer retains sole root access to cloud servers. This means our employees don't have visibility into cloud servers in the way you suggest. Further, as clearly stated in the blog, the issue raised is about data leakage i.e. data being accessible between cloud users. There are other issues regarding vendor security but this doesn't negate the points being made about data leakage.

Finally, your point regarding the encryption being a placebo is specious. There is a big difference between making systems secure against casual data theft/leakage or the actions of rogue employees and a company that is institutionally set up to lie and steal their customers' data. If you think your vendor is actually of that nature then no security measure can help and that's the case for any company you have dealings with. It really isn't a valid criticism of any security measure that may be put in place.

The measures we outline and have implemented on the vendor side do address the real issue of data leakage that occurs with block storage devices in IaaS clouds; they are effective and they are convenient. Our storage performance is generally higher than many other vendors to begin with and we have much feedback from customers regarding this even after using encryption. These customers are getting good performance in a secured cloud. For them it makes sense.

Best wishes,

Patrick

I can't help it, it just doesn't make sense to me.

The only people it would theoretically protect me against are your people (those with physical access). And if I don't trust your people then why should I trust your encryption?

The issue of data leakage between customers (who don't have physical access) seems so trivial to prevent to me that I'm honestly wondering if I'm missing something fundamental here.

What's wrong with just making a loopback file and mounting that to the customer node? I had in fact assumed that this is how most clouds do it (for sanity reasons alone, security being the bonus). Hence I'm rather baffled by the whole claim of "data leakage through reading the raw volume".

Has that ever happened on any cloud provider?

The blog post looks at three main aspects to data storage: - keeping data private (to you as a user) - thinking about legal issues of location and control - thinking about migration issues

Of the first point we outline how data leakage is possible in IaaS clouds. Some may already have measures in place to prevent this. We have our own measures too, some private others public. We offer encryption also as a free and convenient way to secure your data. This has nothing to do with securing physical access which is a totally separate issue. It relates to how customers secure access to their data. As outlined previously we don't have root access or file system level visibility into cloud servers in the way that other vendors generally do (although there are exceptions). As such it does pretty much come down to securing physical access. That isn't the case on other platforms for sure.

Here's another article that you might like (short but sweet) which actually talks directly about EBS and others and the problem of 'data remanance' in a way we can't as a competing vendor:

http://elastic-security.com/2010/01/07/data-remanence-in-the...

Here's an interesting quote:

"The technique of overwriting file sectors does not work without the collaboration of the cloud provider. You are not given access to the physical device, but only to higher level abstractions like file-systems (e.g. Amazon EBS) or key-value based APIs (e.g. Amazon S3). "

I'll get back to you on the loopback technique once I've spoken with the relevant storage guys in our company for feedback.

Kind regards,

Patrick

> How does one revert this without physical access to the drive?

I know this was a rhetorical question, but I'll answer anyway: It isn't possible. Not only is there no way to read latent data normally from a drive that has been zeroed (drives that fail this test are called "defective"), but it is currently understood that recovering data from a modern drive that has been overwritten with a single pass of random data is impossible at any expense.

http://www.cs.auckland.ac.nz/~pgut001/pubs/secure_del.html#E...

However, data can still leak out of cloud stores in the same way that it leaks out of solid-state disks and even magnetic disks: there's no guarantee that a given logical block will always be mapped to the same underlying hardware. A mirrored drive may be fail and thrown in the trash with data still on it, or written blocks might be mapped to different places in an array for any number of reasons. This shouldn't result in leakage to other customers although it is up to the vendor to make sure this doesn't happen.

Depending on the implementation, vendor-supplied encryption may or may not mitigate this risk, but customer-supplied encryption always will because the customer knows where the dividing line stands.

That's great information and you are right. Secure disposal of drives by the vendor is of course also extremely important.

For those deleting virtual drives in the cloud securely the points made in the post might seem obvious but I believe most users in the cloud don't undertake such measures. That's why the encryption option is another way to go and implicit so much more likely to be taken up by cloud users.

Customer side encryption is great and of course usually means access is restricted to the customer, the issue is server restarts, crashes etc. which require manual intervention to get the file system or data directories back up and running again. In a dynamic cloud environment this can be particularly cumbersome.

Best wishes,

Patrick

If I were writing a block provisioning system I would keep a map of what blocks have not yet been written to and always read zero for those blocks. Then the first time it is written, allocate a block that has already been zeroed out. This way I don't waste time and electricity zeroing entire volumes before they are allocated or after they are deleted because there is just one pool of same-sized blocks to draw from. Choosing the block size to strike a balance between allocation delay and cost of blanking is an exercise left to the reader.

That's a possibility, the concern is the performance implications, particularly latency involved in operating such a system which needs to interrogate every data access. The question is whether such a system would have more of a performance hit than simply encrypting (with the added advantages that has anyway).

Best wishes,

Patrick

FWIW, non-block storage services (like Rackspace Cloud Files and S3) should not be vulnerable to these info leaks. I cannot speak to the S3 backend, but this sort of attack would not be possible with Cloud Files. Of course, the use case is a little different when you don't have access to a block-level device.

Yes we believe that's the case too however like you say, object orientated storage services are a different use case to block-level storage devices. Customers running their own databases for example can have these data leakage issues if they aren't careful.

Best wishes,

Patrick

Do you guys know what the situation is with GoGrid? I've been using them for about 6 months now but I've not been using encryption. Am I exposed to data leakage in the way you outline in your blog post?

Any service that reuses the same physical drive space should be vulnerable unless they either store encrypted or do secure deletion. Having used secure deletion before on dedicated hardware it is really hitting the drive performance. That is why I am guessing vendors don't use secure deletion. Using encryption overall looks more efficient overall.

Secure deletion on physical spindles is extremely resource heavy especially if someone is deleting a very large drive.

Storing encrypted first does affect performance but it is generally much less and more predictable (doing a big secure delete on a drive inflicts an immediate and unexpected hit on that part of a storage array).

We think encryption is also just a lot more robust. If you don't lay down the data in the first place in readable form on the physical drives, its eliminates a lot of data leakage possibilities.

Best wishes,

Patrick

Sorry, but this is really bordering on FUD here.

Have you ever looked at a freshly mounted EBS volume on EC2? It shows all zero's for me. And I'm almost sure that was not just a coincidence for the volumes I looked at.

Moreover there are more efficient ways than encryption or "full sweep overwrite" to address this at the storage-level.

Our post is regarding IaaS clouds in general and has nothing specifically to do with EC2 or any other particular vendor. It may be the case that EC2 does have secure measures in place; I wonder how many of their customers can articulate them (or customers of other IaaS clouds for that matter)? We are merely raising an important issue. You can look at the many other posts we have on the other various aspects of security in the cloud of which this post forms the latest instalment regarding data storage.

"there are more efficient ways than encryption or "full sweep overwrite" to address this at the storage-level."

Your suggestion would be?

Kind regards,

Patrick

Your suggestion would be?

See my reply above about using a loopback-file.

It's a one-liner:

  dd if=/dev/zero of=/vols/customer123/vol234 seek=$SIZE bs=1 count=1

There's surely quite a few other ways to skin this cat (eventually arriving at the multi-tenancy features in proprietary SAN appliances), but this seems the most obvious one.

Vukk,

The issue there is more about tracking. You'd have to track every single block and return zero for any that hadn't been used/altered since drive creation. I'm guessing it would prove pretty costly in terms of latency for drive access after initial drive creation but its a good idea potentially. I'll pass it onto our technical guys as well to ask about its feasibility.

Best wishes,

Patrick

Since we are not directly accessing the HDD, couldn't the virtualization layer say if(disk block not written before) return 0; and thus the customer wouldn't have to worry? Or is this too inefficient?

Reply on news.ycombinator.com