Encryption at Rest: Whose Threat Model Is It Anyway?
scottarc.blog
scottarc.blog
When someone borgs[1] up their data to store at rsync.net we just assume that we are the threat. Of course I don't believe that but it's perfectly rational and we encourage people to think of rsync.net as a threat as they design their backups.
Comments in this thread are actually discounting the threat of Amazon personnel, GCS personnel, etc., as if that threat was zero.
Not only is it non-zero, I would go further: if you're storing the data on AWS and generating your keys on AWS and managing your keys with AWS ... you're doing it wrong.
> if you're storing the data on AWS and generating your keys on AWS and managing your keys with AWS ... you're doing it wrong.
This is a reasonable thing to do if you've decided that you trust AWS and expect any violations of this trust to be dealt with by the legal department.
It's less reasonable if you're concerned about AWS employees going rogue and somehow breaking the security of KMS without anyone else knowing.
It's even less reasonable to do this if you're concerned about AWS credentials being leaked or compromised, which in turn grants an attacker access to KMS (i.e., a government would be more successful by compelling IAM to grant access than they would trying to subpoena KMS for raw keys).
(Sure, you can audit access via CloudTrail, but that's a forensics tool, not a prevention tool.)
But that's kind of the point I wrote in the article, no? You need to know your threat model. You've stated yours succinctly, and I think it's a commendable one, but many enterprises are a bit more relaxed.
That's the least of the concerns. Remember AWS is subject to court orders of all types (legitimate ones and NSLs). Even if nobody goes rogue, any data that AWS (or any cloud/SaaS provide) could access, must be assumed to be compromised.
I would agree that if you have no use-case for cloud-side data processing, then the cloud doesn't need the encryption keys; however there are a lot of use-cases for data for which processing is highly advantageous.
For example, if you just want to store data in the cloud, sure, use client-side encryption. But what if you want to query it with, using an S3 example, S3 Select or Athena or load it into a data lake? There are a lot of things one might do. It's useful to be able to query data from within the cloud – without having to transfer it out and separately decrypt it.
This use-case is likely one of the primary reasons why customers are willing to trust cloud providers to manage the keys too – for many data sets, though not all. And that's what technologies like cloud KMS are good for. Access to the actual keys is tightly limited within the cloud provider, both among employees and technologically, and not even other cloud services are capable of accessing them (either accessing the keys or encrypting/decrypting using them) unless you grant them explicit permission, with internal technology that enforces this.
Maybe you didn't intend on using "new fancy foo service" to process your stored data yesterday, but today it sounds good, so you can alter an access control grant and allow it to interact with your existing stored data and the necessary encryption keys. Maybe every time some new data arrives in S3, you want to process that data with Lambda and synchronize it with another system. Then it's helpful if you can grant Lambda permissions to decrypt the data so that it can process it :-)
Generic server-side encryption does do something: it protects against threats that exist at the data center layer, such as certain attacks through lower layers of infrastructure that don't necessarily involve a full compromise of the machines involved, or inappropriate disposal of old storage devices, mistakes by employees, etc. It protects against some insider attacks, just not those by capable administrators.
Most use-cases require 'processing', otherwise there would be no point of most cloud services and we would only ever use, say, S3.
The moment you spin up even a virtual machine, it will need to be able to decrypt the EBS volumes you have attached to it. Even if you encrypt the filesystem with your own mechanism and feed it the key by hand every time, it is still available in the instance. Instance memory is not easily available to AWS employees, but neither are the physical media that backs EBS volumes(whatever it is, details are scarce and all pieces may not even be physically in the same building). If we are concerned to that extent, then we can't use any other service. Which is ok, there are plenty of business that have requirements that disallow the use of any cloud providers.
KMS is a money printing racket for AWS. It only really helps you if someone walks out of a data center with your hard drive. For everything else the attackers are just going to get your IAM creds which have open access to all your keys.
Either way, Thank you for providing such an affordable service !
https://rsync.net/resources/howto/borg.html
… both of those are very concise (especially the bsd one) and, while they can be used as a general how to, they have rsync.net examples.
Related: I have, personally, recently started to use ‘borg mount’ and it is fantastic.
Thank you for this comment, this continuously blows my mind.
If one hands over cleartext data to some third party, one must assume they will abuse it. They might not, but "might" is not a convincing control in a threat model. If they can, one must assume they will. And plan accordingly.
Sure, sometimes it is ok. But any threat model that assumes a third party that could abuse the data but will never do so, is flawed.
It is vital to remember two things: Even if one 100% trusts the current management of a company to do No Evil, management can change. Second, regardless of managment, a company is subject to subpoenas and NSLs, so your data will be handed over.
If you don't directly control the generation, storage and usage of all encryption keys, your data is effectively in the clear.
I've work with physical servers in many compagnies. Every time, storage devices have a lifecycle : they are bought new, then plugged (and some data are written), then some times then move in another physical location, then they "die" (= put to the trash, either because they broke or for other reasons).
Encryption at rest is an efficient way to secure data for the latter cases.
The workers stole a hard disk ? No data is stolen.
A hard disk is lost during transit, somehow ? No data is stolen.
The device which is broken is retrieved by someone, opened-up, and being read-at directly ? No data is stolen.
Your old device, put to the bin for some reasons, that could still be read if plugged on the proper hardware ? No data is stolen.
All of that with little performance impact, and no software modification, few engineering overhead, very little work to do. It eases the lifecycle of storage devices, because storage devices are now worthless per-se (except for their physical cost, indeed). They carry virtually no data, no worth.
Can you propose any other way to protect against those thread models ? Rewrite every software, so that every programs handle their own private keys ? Yeah, that's a nightmare, not gonna happen. And even if it did .. how would you encrypt your rootfs ? Ha yes : encryption at rest :)
Which is also what the author write:
>If you’re only interested in compliance requirements, you can probably just enable Full Disk Encryption and call it a day. Then, if your server’s hard drive grows legs and walks out of the data center, your users’ most sensitive data will remain confidential.
> Unfortunately, for the server-side encryption at rest use case, that’s basically all that Disk Encryption protects against.
(Something tells me that you did not read the article that is so narrow minded)
But yes, if the author drops all security thread related to encryption at rest, then encryption at rest is useless. I agree.
The point of my article was not "Encryption-At-Rest Is Bad" as you seem to have taken it to mean.
Rather, the point is that other techniques, when you sit down and actually think them through, do not provide any significant value over just phoning it in with Full Disk Encryption.
How you get from "software libraries that encrypt-at-rest routinely fail to provide value on top of full disk encryption" to "Scott says FDE is bad" is unclear.
Additionally: From a software perspective, the risks you all outlined are morally equivalent to the hard drives grew legs and walked because they are the same risk; namely, loss of control of the actual hard drives.
The article in question is focused on threats when those drives are plugged in and the keys are being used to encrypt/decrypt data. As several others have pointed out already, I explicitly state this, repeatedly. I don't know how to make it more clear.
So not only are you not adding anything to the article. You actively try to dismiss that the author has thought the cases you bring up through (and calling them out as narrow minded on false grounds).
I think we all would love to see a risk that is mitigated by encryption at rest and is not already being mitigated by disk encryption.
Indeed, they said exactly that. See:
>Important: I’m chiefly interested in discussing one use-case, and not focusing on other use cases. Namely, I’m focusing on encryption-at-rest in the narrow context of web applications and/or cloud services.
Further, if data is encrypted at rest then there are multiple levels of auth that must fail for a breach to occur, namely access to the data and access to the key.
>In web applications and cloud services drives could still be misplaced, stolen, or improperly disposed of.
This is explicitly called out in the article by the author, despite it not being part of the threat model the author is examining. And people are still bringing it up like some sort of gotcha.
See (again):
>This is not a comprehensive blog post covering every possible use case or threat model relating to encryption at rest.
Regardless, even in their very restrictive scenario, it provides defense in depth as I said.
Loss of control of the hard disks may have many different ways it can manifest in the real world, but from a cryptography and software development perspective, is congruent to other flavors of the same underlying problem.
That's not being "restrictive", it's recognizing the common denominator.
It doesn't need in-depth discussion, and the way data is still highly exposed despite disk encryption is very important, but that implication is not great.
Then what is the cost / overhead / complexity / other cons of adding encryption at rest? Cyber budgets often go crazy I see so many clients that are buying tech based on marketing hype or the security teams lusts for cool tech rather than what reduces the risk the most for the dollars available.
The option triggered the implementation of a dm-crypt layer between the physical device and the upper storage layers. Crypto keys were stored in the storage system. Once revoked, the whole server was rendered useless (from a data thief point of view).
We benchmarked the stuff a bit. Indeed, there was a loss. While dm-crypt uses AES (with hardware acceleration) and since we had multi-hundreds of thousands IOPS per device, we did not care.
I don't think the vast majority of people with IT/ops experience seriously thinks that encryption at rest provides data protection from people getting unauthorized access to the system, in person or remotely...aside from maybe management that ended up in charge of IT and engineering departments without almost any practical-skills background in either.
The author is confusing "it costs us nothing (now that encryption can be done in hardware and is integrated into most desktop operating systems) and protects in some scenarios, so yeah, we just decided to mandate it always be done" with "PEOPLE THINK ENCRYPTION AT REST IS A MAGIC BULLET LOOK AT ME I'M INSIGHTFUL, POST LINKS TO MY BLOG ON LINKEDIN!"
The whole post is insulting to the intelligence of even a fairly junior desktop support technician.
What in the article gave you that impression?
I do not hold this confusion in my mind, nor did I deliberately encode such a statement in my blog. I'm curious why you think this is what I was saying.
> The whole post is insulting to the intelligence of even a fairly junior desktop support technician.
If that was true, every time someone posts "Show HN: My Hot New Database Encryption Library in Haskell", they would be mitigating the confused deputy attack by design, rather than what we see today: Namely, failing to even protect against padding oracle attacks.
That's what the article was actually talking about.
Seems to totally miss the point, but I still upvoted because it illuminates the purpose of disk-level encryption so well, adding color to the conversation.
But secondly, the idea that anyone who actually has a say in if the data is turned over (the legal team, executives) give a flying fork about how a random sysadmin in their data center feels about it is wildly off base. The data is going out or not according to legals instructions. Either sysadmin Bob does it, or he takes a principled stand, gets fired, and Bob 2 takes care of it.
Most of my colleagues (at least those I've met) tend to be conscientious about data access and privacy. I host several E2EE services for friends/family because I believe in privacy and data sovereignty. But if my company's legal department says "give access to <these guys>," as a sibling commenter said, if I refuse then I'm fired and those guys will still get access. So I do what legal says and keep my job, TYVM.
If you use a cloud service and they have an encryption at rest feature that you enable, the default is for them to control the key. Or, for Azure, put a "customer controlled" key in Key Vault. But again, that's their service. In this scenario you're only protected against people physically in the data centers not their DevOps folks. But, that feature gives you a checkmark for SOC2 or other regulations...
The problem protecting against DevOps folks at a cloud is: 1. You are burdened with putting a key somewhere outside of their cloud. If you do something at the filesystem level like enable bitlocker on a VM you're going to experience pain during reboots and it's not possible on a root volume if you're not given a console. 2. You can't do this on a cloud service like RDS. You'd have to do row level encryption with your application doing the decryption/encryption. But, your application has to have the key and now your back to #1. The VM with the key needs the root disk encrypted or the drive where the key is store encrypted. And again, you're not able to use a cloud service like EKS or App Service you're stuck with VMs.
Generally, I tend to just stick with regulation requirements which protect against the physical hard disk.
Arguably, customer data should not leave a secure location on a physical disk. If you want to move your server, backup the data, wipe the disk, move the server/disk/etc, and then put the data back on the disk once it is back in a secure location. The author mentions this in the article - define a secure boundary, don't let the data exit that boundary unencrypted.
Okay, let's argue. Under what threat model is it a problem to transport sensitive data on encrypted disks?
Arguably, your position is bogus. Why cause a bunch of extra copying around to move a server. "Back up" indeed - to what? Another disk? It's turtles all the way down.
I don't. I know it has amazing bandwidth and it might be a lot faster than network transfer.
> Arguably, your position is bogus.
That's not nice.
> Why cause a bunch of extra copying around to move a server. "Back up" indeed - to what? Another disk? It's turtles all the way down.
Yes, to another disk, in the same data-center or in another one. If the data is encrypted end-to-end, it might be better to transfer it to another data center than to move the data between data-centers on physical disks for security reasons. It's less likely that an adversary is able to execute an attack where it is able to pin-point the exact moment and the exact data they want is being backed up between data centers. On the other hand, organizing a heist of a few trucks filled with physical storage leaving the data center might be easier to pull.
In these two scenarios we expect the encryption to be equally hard to crack for data at rest and data in transit, which is usually not true for performance reasons.
Serious question: is there a sane, working, single solution to ensure that ? Flash-based devices are so complicated (versus the old rusty devices)
Anyway, this sound exhausting
PCI 4.0 disallows using Full Disk Encryption to fulfill its encryption at rest requirements, because it demands a finer-grained encryption authorization model. For applications subject to PCI 4.0 compliance, my clients have to encrypt beyond the application level, and down to the service account level used by the application (most of these applications use multiple service accounts for representing teams that use the applications), with different files encrypted for different authorization groups.
There are very few enterprise scale solutions addressing this at the moment, and the stopgap I've seen is indeed the application teams are resorting to handling their own keys if the enterprise doesn't offer a finer-grained solution. The QSA's don't yet understand how seismic a shift this is, nor how inadequate the assumed authorization schema is for the many edge cases out there that will be unable to adopt the assumption a single key for a single group is sufficient to model all use cases. It will be a little messy until folks sort through the edge cases coming down the pike.
PCI 4.0 demands more granular encryption authorization models, where data is either encrypted at the file or column level, or access to plain data is restricted to more specific roles than just the OS user account. This is to reduce the risk of unauthorized access.
There are a few dev-centric data protection solutions that address these requirements seamlessly. These solutions allow you to store and encrypt any type of data with simple APIs while transparently managing keys, rotation, and granular access controls.
I am currently evaluating Piiano Vault (https://www.piiano.com/pii-data-privacy-vault) as one such solution for a new product that I am currently working on that would require a PCI compliant zero-data solution. Seems like its delivery model is very flexible - it can be fully self-hosted or consumed as a managed SaaS, providing robust data protection tailored to my specific needs.
Will update with my findings!
i think you misread the article. i found it to be a great, even excellent, discussion of WHY and then explaining HOW in terms of the WHY.
Instead a lot of motivating factors I've seen are about preventing various paths for "legitimate" data disclosure to third parties. For example, when data at rest is combined with additional requirements like "bring your own key" it means a subpoena or NSL needs to be served on the _first party who owns the data_ (as they need to provide the keys) and can't be served on just the cloud provider without the first party having at least visibility of it.
Encrypting each field with a distinct key that an attacker cannot glean by simply exfiltrating all the data on disk and/or all the data in RAM protects against online attacks in a way that full-disk encryption cannot.
The real question is: does Salesforce do this properly?
I can’t speak for the implementation, but my guess is that it’s been very thoroughly vetted by both internal security and external pen tests. They wouldn’t market a high profile security feature without that.
(1: I am an ex-Heroku / Salesforce employee)
Anyways, the improved security comes from the fact that even when the server itself is improperly accessed (maliciously or not), the data you aren't currently accessing remains encrypted.
With (just) full disk encryption, you aren't protected when the (running) server is accessed. All of the data can be exfiltrated in plaintext.
Encryption "at rest" in the database prevents someone with server or direct db connection from pulling the data.
I had never really thought of those as two different vectors, but of course they are. Thanks for clarifying!
With Salesforce and how a lot of these companies manage their security model, I'm still confident that investing in securing unauthorized user access is still orders of magnitude more useful than putting time and effort into this vector.
These are addressing two different scenarios, so they should be mitigated separately. In one case, you are mitigating against unauthorized access. In the other, you are mitigating the damage that can be done when someone has already gained unauthorized access (however that occurred). After all, the only system immune to unauthorized access is the one that doesn't get powered.
"Defense in-depth" is thrown around a lot, but it really is important. I do agree though, when it comes to priority of implementation, I would start with protecting against unauthorized access first.
[0]: https://learn.microsoft.com/en-us/sql/relational-databases/s...
This is a reasonable limitation when you're aware of the attacks on Order Revealing Encryption: https://blog.cryptographyengineering.com/2019/02/11/attack-o...
I worked on a project where separate databases (a la Postgres) tied to separate clients wasn't enough. Postgres still can read across the databases.
With TDE we tie the key to the individual clients meaning even if the engine messes up, since the connection isn't made with the right key, you still can't read the contents.
We still did encryption at rest as that comes for free these days, for reasons mentioned here in the comments.
I just wish Postgres would come with TDE. Paying for software is fine, but the cost of MSSQL is way more than $0. In fact, it's usually cheaper to set up a Postgres instance per client. That way, when the engine messes up, well, there is only data of that client.
And I know a database is unlikely to mess up. I'm more likely the culprit, and as such I prefer to have my guardrails.
Originally, I never felt that the EAR issue was that germane at data centers, since you're mostly protecting against someone backing through the loading door with a truck and stealing trays of drives.
The disposal issue is valid, that was just something we were diligent with using a secure disposal service.
Key management has always been an issue. Since no one wanted to be there when the machines spooled up after a glitch at 3am to type in a password to open the volumes. Everything else is basically locking the file cabinet drawers and taping the key to the back of it.
Nowadays, it's different. Its more ubiquitous. Encrypting a laptop is a mouse click, and painless after that. Cloud providers have the infrastructure to manage it at their level.
I'm still now sure what the solution is for a "self hosted" infrastructure. Hardware key modules are still pretty expensive. I haven't (not that I've looked at all recently) seen a writeup of how best to set of encryption on those 4 Dells my friend has racked across the country in Georgia (though even modern machines have some level of volume encryption I think).
There are two audiences for this articles then, the ones who know what they don't know (or know it all), and those like me who are ignored-squared.
I found it extremely helpful to help me "know what I don't know". If you have no idea what "encryption at rest" is or why it is important, then this is very useful and helpful.
As the author clearly states it does not cover other things you don't know, like why and how to use full disk encryption.
That said, it would be helpful for us i² [i squared] folks to have some of the more basic terms explained. Although "encryption at rest" is somewhat understandable, it would be pleasant to have it explained. For example what are the other kinds of encryption that are not "at rest"?
There are a bunch of diligent amateurs out here, ones who know we don't know what we don't know and are attempting to build cryptographic things. We learn not to create our own implementations for example, but articles that specifically address us are good.
[edit for clarity]
That's helpful feedback, actually. What terms seemed opaque or misleading to you as you read it? I'm always happy to fix mistakes in blog posts to improve clarity.
> For example what are the other kinds?
I contrast "encryption at rest" with "encryption in transit" (i.e., TLS) and "end-to-end encryption" (E2EE for short; i.e., what Signal gives you).
This is a crypto nerd blog post though. The whole point is to talk about cryptographic library design!
"You can get this property by stapling HKDF onto your protocol (once for key derivation, again for commitment). See also: PASETO v3 and v4, or Version 2 of the AWS Encryption SDK.
It may be tempting to build a committing AEAD scheme out of, e.g., AES-CTR and HMAC, but take care that you don’t introduce canonicalization risks in your MAC."
I would almost suggest breaking stuff like this into two articles, one which is very technical and correct, and one that conveys the high-level message. The high-level one can link to the technically correct one whenever the urge would come to explain something more fully.
I actually did learn quite a bit from your article. The use of tech terms like HDKF & AEAD was helpful rather than a hindrance. For example the phrase "stapling HKDF onto your protocol" is surprisingly helpful. I looked up HKDF, and the use of "stapling" gives me a concept of how it is used. So good.
Going back over it, I believe the real problem is that your previous post, "Lucid Multi-Key Deputies Require Commitment" is required to reading (or knowledge) for this post. Once I read that much of this was easier. You hid that reference is in the "why should you believe me" and yet to my reading the this article builds on that one. You define the terms and provide a context that is missing.
So concrete suggestions (easy to come up with after the fact!):
- ask yourself if this is a continuation of a thought, topic, etc and give refs if so. - in addition to "why you should believe" maybe add a section "for i²" readers :-)
Cool things are ones where you never see the world the same. Just to reiterate, I don't see the world of cryptography the same after reading your article, despite the quibbles, so thanks.
And especially thanks for taking the time to share your experiences and observations with me. That's how I improve as a writer.
Basically I am complaining that you choose to rephrase a perfectly good description of a problem (with a lot of history) into an incorrect statement.
So it would be more accurate to say that “there are some things that I don’t know that I don’t know.” However, “unknown unknowns” and “known unknowns” are more succinct. And, thanks to that idiot Rumsfeld, more commonly understood.
I was cross country skiing across a frozen lake. The ice was 24 inches thick. I knew it was that thick. I did not know that where a lake narrows there can be current that prevents ice from forming. Moreover, I was unaware that there were major issues about frozen lakes that I was unaware of. I was not aware there was any risk and therefore came much to close to falling through the ice. Those little "crack crack crack" sounds stick with me. There have been other examples when I was unaware that I was ignorant of life threatening issues.
My use of that phrase is perhaps particular to the fact that most of my realization of "unknown unknowns" has been when they are life threatening.
People asking about "encryption at rest" are really asking if backups of your web application data are encrypted.
Earlier I think it was quite a plague when just un-encrypted backup files were leaking out because someone exposed them on some open FTP to "quick and dirty" copy backups to new environment or to some test environment - and forgot to close it down or remove the files.
Other threat would be developers exposing database server directly to the internet because someone from marketing wants to connect "new shiny super business intelligence" and developers not knowing better than "allow all" on firewall, then someone might steal raw db files but might not really have access to web application and encryption keys.
For the reasons mentioned by author I can see how it seems like security theater. But I think my reasons are quite valid and on topic.
The thing that's security theater is encrypting insecurely, or failing to authenticate your access patterns, such that an attacker with privileged access to your database hardware can realistically get all the plaintext records they want.
Security theater should be defined as:
Doing things that outwardly appear to improve security but have de minimus or less effect on actual security.
The 93 section questionnaire from bigco's IT department is security theater. Filling it out does zero to improve security for bigco or myco or my users.
Right. And that's exactly the situation the article describes.
The accusation of "security theater" was only levied when IT departments reached for the "full disk encryption" potion to mitigate the ailment of "attacker has active, online access to our database via SQL injection", when that's not at all what it's designed to prevent.
They can insist that they're "encrypting their database", but does it actually matter for the threats they're worried about? No. Thus, security theater.
The same is true of insecure client-side encryption.
But once the questionnaire mattered, they started doing it just so they could legally answer "yes" to that question. Things like finally changing the default admin passwords on that service they installed a year ago, and testing backup recovery to find out that it actually can't be done due to a bug in the backup script skipping some key data.
Once contract with big co was on the line I got permission to do security and do it good.
Even though 80% of questionnaire was not applicable it still did the good job.
The problem of course being, most health care breaches are on applications that aren't at rest so all the data was being stolen anyway.
From 2012-2021 I worked in health tech and on many calls with large customers and security questionnaires were on whether we were actually storing their data encrypted at rest. We even had to get audited for Aetna to validate encryption at rest (amongst other things). To me this seemed like such a joke of a requirement because all our data was in AWS, and breaches were far more likely from other avenues.
So to me this reads as a jaded SWE or CISSP who has dealt with how much attention this one attack vector is paid, but ultimately it is kind of a given now in modern cloud infra.
No, I work in applied cryptography.
When I joined Amazon, the team I was hired on was called AWS Crypto Tools (which owned the AWS Encryption SDK, among other developer tools), while another team was called Transport Libraries (which owned S2N).
When I left in 2023, they started adopting more of the "Encryption At Rest" lingo for what was previously called Crypto Tools. I don't know if they landed on different verbage since then.
> after HIPAA passed encryption at rest was the buzz word of the decade as it was one of the primary requirement of HIPAA.
Interesting. Thanks for sharing.
I was hired in 2019 as a fully remote employee. This means my options in 2023 were a) move to Seattle or b) walk. I chose to walk.
The leadership of the Cryptography org fought tooth and nail to get an exception for me, but were unable to do so. I still hold everyone in AWS Cryptography in high regard.
What benefit does encryption at rest solve for something which is never intended to actually rest?
For example, a MySQL database powering a web application is expected to be alive and responding to requests 24/7. It's never really intended to be at rest.
So what benefit does encryption at rest bring? Won't a hacker be attempting to take data when it's online (and therefore not resting)?
I just don't get the value of it for always online servers.
You can remove storage devices from online servers, without interruption. Said devices will contains data that could be "lost" that way. Hence: encryption at rest.
I haven't read the post yet, but I feel like expecting a perfectly well-defined model is kind of in bad faith. Defense in depth is established, FDE is one tool among many.
By focusing on the hacker we forget about the person who may get the equipment downstream; procedures/processes fail, and so on.
It's preparing for the unknowns. Sounds like paranoia? That's the job! Defending against human nature - malice, forgetfulness, etc.
I guess I'll close with this: you're the only one who can make your security assessment. What's important, what's at risk, and so on. It's trade-offs all the way down.
I am pretty sure your database stores things into /var/lib/mysql : this is what would be encrypted (without support from the software)
So, if you lose your SSD, nobody is able to read your database's content
That's the point of the article, luckily enough. "Disk Encryption is important for disk disposal and mitigating hardware theft, not preventing data leakage to online attackers."
> So what benefit does encryption at rest bring? Won't a hacker be attempting to take data when it's online (and therefore not resting)?
Yes (again, illustrated in the article).
The first important part is: Encryption at rest protects higher levels of the stack from access by lower levels of the stack. If your attack is working at an application level - e.g. you have a database connection - encryption at rest is no tool to deal with this. Encryption at rest is more about protecting the database from an attacker with physical access.
The second consideration is: There will always be a tradeoff between availability and security when dealing with encryption at rest. Manually decrypting a system is very secure, but if that system goes offline at 3am, it'll be offline until the device is decrypted. Automatically decrypting a system using Clevis/Tang, TPM, Bitlocker and such gives you more availability but could make it possible for an attacker to access your data if they have sufficient control.
And that's the third consideration: If an attacker has sufficient control, they can defeat automated decryption of encryption at rest, and they may have ways to start attacking manual decryption of encryption at rest as well. Like, you might be able to start looking at memory contents, disk writes and start doing some differential cryptoanalysis to attack encrypted data and such.
But with all of these three together, you arrive at the goal and security level we have formulated for our encryption at rest:
Our encryption at rest is supposed to defend us against employees of our hosters getting access to one or two of our virtual or physical storage devices. If they have access to one or two storage devices, they must not be able to access customer data on the devices.
Naturally, the question is: But what happens if they have more drives?
Well, the simple answer from the whitepaper is: That's a problem for the lawyers.
Handling 1-2 of our drives is entirely arguable as daily business. Swapping drives via remote hands or dismissing dedicated servers with 2 drives at a specific hoster happens a lot and then they handle 1-2 drives, and our goal ensures they cannot gain access to customer data then.
However, if a datacenter tech starts pulling 3 or more drives and starts analyzing data on them together, that's outside of normal operational procedures and we can start considering that an attack and start sueing them. Or they are being directed by law enforcement. Both are issues for the legal teams though.
At least that's our view. Encryption at rest works in a very different set of circumstances and mindset than other security topics in a software stack.
This is of course the beef. What's the best practice in managing user data keys so that data is available only when there's an authenticated user around? There are ways to derive keys from the secret exchange involved in user authentication.
What does it mean for an authenticated user to be "around"?
If you want a human to manually approve the decryption operations of the machine, but it can still store/encrypt new records, you can use HPKE so that only the person possessing the corresponding secret key can decipher the data.
At least, you can until a quantum computer is built.
I wasn't thinking of manual approval of any cryptographic steps. Just that when you log in to work on your data stored in the system, the system can only then decrypt the data, and when you log out, the system forgets the keys until next time.
It all depends on the type of app of course.
I would need to actually sit down and walk through the architecture, threat model, etc. to recommend anything specific. I'm not going to do that on a message board comment, because I probably am missing something.
Decrypting on the "client" (app server) means you can't really use its native query language (SQL) effectively on the encrypted columns.
Not sure what the state the art is in searchable encryption for db indexes, but just trying to do stuff that requires a scan becomes untenable due to having to read and decrypt on the client to find it or aggregate it.
The security sandbox becomes app server memory instead of db server memory preventing effective use of the locality on the db server. I didn't see the article address this.
Can make sense to encrypt specific sensitive columns that are not used for searches or aggregations later, but many systems the reason you have discrete data columns is to query them later not just retrieve a single record to decrypt and view on screen vs just storing a single document and encrypting / decrypting.
Disk encryption is easy to use without reducing the functionality of the DB, client encryption specifically and purposely handicaps the functionality of the DB so its use case is very narrow IMO.
I tend to treat both the app server and db ram as unencrypted so they require good access controls to use them (don't let Bob run sql queries against all the data unless he is authorized to do so).
On the one hand, you can require that 12345 always maps to AWERQ, in which case an attacker can use frequency analysis, metadata chaining, etc. to determine with some confidence that AWERQ = 12345. Calling this "pseudo-anonymization" is definitely more accurate than calling it "encryption", but you might as well just use a one-way hash function instead. It doesn't do anything against determined attackers with prolonged or broad exposure to the data; I don't see the value except perhaps for compliance with poorly thought out or outdated regulations.
On the other hand, you can require that 12345 always maps to a different string every time, but that means you need a different key/salt/IV/nonce for every row or cell, defeating indexing and aggregation, and so all queries become full table scans. This significantly frustrates an attacker, but also significantly frustrates legitimate operations.
Zipcodes are short. If not given extra padding, their ciphertexts will still be short.
Zipcodes are also low cardinality. Unless you use multiple salts/nonces/IVs/keys, the frequency of ciphertexts will match the frequency of plaintexts.
In many situations, a prepared attacker will be able to insert their own data beforehand, allowing them to perform a chosen-plaintext attack and potentially decipher much of the data. The best protection against this is to not reuse salts/nonces/IVs/keys and thus again defeat performant searches.
None of this is to necessarily say it's not worth it but rather to keep in mind the article's point: know your threat model.
There are a lot of different approaches, but the one CipherSweet uses is actually simple.
First, take the HMAC() of the plaintext (or of some pre-determined transformation of the plaintext), with a static key.
Now, throw away most of it, except a few bits. Store those.
Later, when you want to query your database, perform the same operation on your query.
One of two things will happen:
1. Despite most of the bits being discarded, you will find your plaintext.
2. With overwhelming probability, you will also find some false positives. This will be significantly less than a full table scan (O(log N) vs O(N)). Your library needs to filter those out.
This simple abstraction gives you k-anonymity. The only difficulty is, you need to know how many bits to keep. This is not trivial and requires knowing the shape of your data.
https://ciphersweet.paragonie.com/security#blind-index-infor...
I proposed the same technique to AWS, who adopted it under the name Beacons for the AWS Database Encryption SDK.
https://docs.aws.amazon.com/database-encryption-sdk/latest/d...
That's awesome that AWS has now rolled it into their SDK. I had to custom build it for our Node.JS implementation running w/ AWS's KMS infrastructure.
Are you the author of the paragonie website? The coincidence was startling. If so, I greatly thank you for the resource.
Edit After going back and re-reading the blog post, looks like you are the author. Again thank you, you were super helpful .
Thanks. Yes, I'm one of the authors.
As someone who has issues remembering where they left their house keys at times, I want this on a bloody coffee cup/t-shirt.
Further, every practicing cryptographer everywhere should be forced to keep one on their person at all times, and be required to present it to do normal tasks as a reminder to them of the suffering their work creates for others implementing their cryptosystems. They probably won't care. But it'd make me feel better if they were as annoyed in the everyday as much as I am wrangling this kind of nonsense.
I've heard this quote a lot over the years, but the first person who said it was probably Lea Kissner, the former CISO of Twitter.
Here's their most recent post of the same statement: https://hachyderm.io/@leak/110784289970982813
The cryptographic details about preventing confused deputy are excellent and worth studying.
> "What’s happening here is simple: The web application has the ability to decrypt different records encrypted with different keys. If you pass records that were encrypted for Alice to the application to decrypt it for Bob, and you’re not authenticating your access patterns, Bob can read Alice’s data by performing this attack."
An interesting question here is whether or not there's a god key that allows the administrator to decrypt all the data even if they can't authenticate as the user (or if they just have copies of all keys). Searching HN for 'lavabits' turns up some results related to this, e.g.
https://www.eff.org/press/releases/eff-has-lavabits-back-con...
The actual facts of whether or not data are encrypted using some whiz bang algorithm are irrelevant as long as some intermediary is ensuring that the data are only accessible by the intended clients.
Sometimes I wonder if all the focus on encryption is actually wasting cycles that could be spent instead making sure the authorization model is bulletproof.
For instance if a DBMS could ensure that only client A can access client A's data then does it matter if the data are stored encrypted in the DB?
You might say, well if they aren't encrypted then anyone with root can just read the data directly but it may be the case that anyone with root will be able to access the data regardless of whether it's encrypted because they can just pull it from the memory space of the DB engine.
There are a lot of considerations but it does seem like people get caught up in the "how" of encryption because of all the fancy maths and cool sounding algorithms rather than focusing on the "what" they're actually trying to accomplish which is usually "prevent clients from accessing data they shouldn't be able to access".
You want to delete data? Toss the keys. You want to confidentially process data? Make the keys available to a TEE or such. You want to prevent yourself from having constant access to the data? Let the client provide the keys. And of course, you want to protect the keys? Use an HSM.
Security compliance and real world security are a universe apart. You have encryption at rest with mandated AES-GCM/SHA512? Cool story bro, some teenagers just broke into your network with a bit of social engineering and a 6 year old CVE.
I find that very easy to belive.
I am confident that they do. Even better, they can be configured to use your KMS key rather than the service key, and you can configure KMS to use external key stores (i.e., an HSM in your datacenter outside of AWS's control, that you could theoretically pull the plug on at any time).
On AWS RDS, you can't turn off encryption once you turn it on.
The encryption secret keys always stay with AWS and all you can download are the public keys.
So I'm not so sure what's the point of encryption at rest in AWS except just to tick off a compliance and regulatory checklist.
The private key is with them anyway, just don't encrypt and save few milliwatts of power.
> The private key is with them anyway, just don't encrypt and save few milliwatts of power.
"Them" is Amazon, a company with over 1 million employees, last I checked.
It's perfectly reasonable to trust the KMS team to keep your keys secure, even if you don't trust the RDS team to never try to look at your data.
I know it's tempting to think of all of AWS as a sort of "Dave" who wears multiple hats, but we're talking about a large company. Protecting against other parts of the same company is still a worthwhile and meaningful security control.
From my perspective, the secret keys I don't have. Just AWS has and they can decrypt whatever and whenever they want maybe because they have a warrant or some three letter agency has them do it.
If the database is live, then the data is able to be decrypted and who knows where it ends up. Encryption at rest solves only the threat scenario where the RDS team has access to the database storage layer. It doesn't do anything to mitigate any threats after it has been read from storage.
Encryption at rest protects against this.
> Unfortunately, for the server-side encryption at rest use case, that’s basically all that Disk Encryption protects against.
If you aren't able to self host, then encryption at rest is a real use case and the next best thing to actually controlling your data. That being said, obviously self hosting with FDE@Rest is the best.
Or you can end up like the people who lost their data [1][2].
[1] https://spectrum.ieee.org/thousands-of-bitcoins-stolen-in-a-...
I don't see how encryption at rest could've changed the outcome.
In the article, the cloud provider, which has full control over the VMs, was compromised. The VMs were hosting various Bitcoin services, which needed continuous wallet access for operation. So, I'd say there was no data at rest to be secured. The attackers could theoretically patch the application to make malicious transactions or just extract the wallet from RAM.
Also, the article suggests that the attackers were getting inside the running VMs rather than accessing VM storage directly.
Edit: Since this has gotten some negative votes, I'll happily expand.
The two primary examples of FDE that are real-world useful (i.e. not just checking boxes) is loss of physical control of a device and cryptographic erasure (at device end-of-life).
Neither of these use-cases in relevant to the threat model the article is discussing, but it's ridiculous to say that FDE is only for SOC.
If you agree that FDE is good for physical access to lost/stolen devices and cryptographic erasure, I'm not sure why you don't think that applies to hardware in a data center which is just as capable as being lost/stolen, and also needs to be securely disposed of.
>But expecting it to do anything else is just magical thinking.
It certainly does more than just check boxes for SOC, which was my entire point.
The quality of healthcare privacy reasoning is like, instead of searching for user Michael Knight in a database of cancer patients, you hash the name, use a bloom filter to determine if the hash exists in the protected data set or not, and then tell your regulators and the public that the foreign jurisdiction research assistants hired from fiverr don't have access to your health information because everything is encrypted and we only compare hashes. it's like that "sudo make me a sandwich," cartoon but, "cryptographically disclose your health information."
Wow, okay, you have me hooked.
> instead of searching for user Michael Knight in a database of cancer patients, you hash the name, use a bloom filter to determine if the hash exists in the protected data set or not
The protocol you loosely described here could be either totally fine or horrendously broken depending on the implementation details and architecture of the application. Not to mention the universal cryptography concerns; i.e., key management.
> and then tell your regulators and the public that the foreign jurisdiction research assistants hired from fiverr don't have access to your health information because everything is encrypted and we only compare hashes.
You've said "hashes" twice now, so I have to ask:
1. Are they only hashing, or are they using a keyed hash (e.g., HMAC)?
2. Are they doing something ridiculous like using MD5?
If you're familiar with cryptography at all, passing around MD5(data) is vastly different from truncating HMAC-SHA384(data, staticKey) to a few bits and protecting staticKey with an HSM.
Without more detail, I can't tell if your assertion that cryptographers are being dishonest is warranted. It's sort of like simplifying RSA encryption to "multiplying integers". Yes, that's what's happening on some level, but some essential information was omitted.
my example is that the scheme uses tools in a way that is meaningless because I can just take a phone book, hash the names, and check to see if they are in the cancer database as though it were an cleartext database. to say the data subject's data is private in this case because it is hashed (or often, inaccurately, "encrypted") is to mislead people who make decisions about it. I'm saying the designers of such systems represent themselves as cryptography and security experts who build weakened systems like this to mislead regulators on behalf of their bosses and users who just want the cleartext data.
most protocols are a shell game where if you don't know where the root of trust secret is managed you're the sucker at the table. my experience has been that working cryptographers (protocol designers) in government, health, and financial institutions in general are a very refined class of bullshitters who pound the table whenever you ask them about details, and that one should not be intimidated by their theatrics. Hence in PHI management, dishonest cryptographers working on behalf of anti-privacy interests are the main threat.
All I can say here is, "Yikes."
If you (or, well, anyone) ever need an honest third party to audit a cryptography design--whether it's because you suspect bullshit or just want assurance that it's actually a good design--talk to the Crypto team at Trail of Bits.
https://www.trailofbits.com/services/software-assurance/cryp...
There is a book about that, "Malicious Cryptography" by Adam Young and Moti Yung. Not about dishonest cryptographers per se, but about various sneaky tricks that could be used in crypto systems.
Anyway I'm sure you remember the notorious Dual EC DRBG.
It does protect against physically taking of and improper disposal of HDs when done correctly. Which can be quite a significant vector.
If the goal is to put “military grade encryption” sticker in the sales deck, or to pass some certification, or to create a vague cloud of plausibility that you’re taking security seriously in some other way, then absolutely it’s security theatre. It deserves to be called out.
But who is stealing data off of servers by taking the server?
Maybe it saves you when you dispose of the disks? Maybe a home server in a break in?
This has always seemed obvious to me.
- Physically moving servers outside of the cages they are supposed to be in. (inside/outside DC) - Easier to dispose of disks (repair/EOL)
Away from pure bare metal, in the presence of bugs in security separation someone could steal data simply by having a VM on the same server as yours. Encryption at-rest with each VM or service/account having their own keys removes the risk of data accidentally becoming visible in another environment that is sharing the hardware.
Though the chance of some useful data being revealed this way is pretty small on large scale cloud installations, and if you are using a cheap VPS host for sensitive data you might not be valuing your data enough.