AWS Secrets Manager – Store, Distribute, and Rotate Credentials Securely
aws.amazon.com
aws.amazon.com
Second reaction: Holy shit that's expensive [for what it does].
ParameterStore is free (minus the KMS component). The only value-add is secret rotation and that's not something that most of the time makes sense to use. [Edit: I'm not advocating for no rotation; see replies]
Edit: Had more time to think about it. Someone enlighten me: What's the difference between writing a rotation lambda for this new product, vs. writing a rotation lambda for ParameterStore that you then cron? The pricing really doesn't make sense.
From a security perspective, you should be rotating secrets somewhere between annually and every 90 days, depending on your business/security/compliance requirements and the nature of the data secured by the secret.
In other words, yes, you should be rotating what you can rotate, but this doesn't always help. Furthermore, it doesn't justify the pricing. It would make sense if this were, say, "Hey, you can now auto-rotate SSM-PS secrets for an extra $0.40/secret".
Right now it just seems weird.
[Edit: I just saw the custom rotation bit of the article. Cool; but if you're at the point of setting up lambdas for the rotation, you might as well cron a lambda on top of ParameterStore...]
("Well-secured workstation" is arguably an oxymoron, of course...)
It's much better practice to use the bastion as a proxy to the other hosts. This is easily achieved using the ProxyCommand option of OpenSSH.
No agent forwarding, no secrets kept in random hosts.
I’m currently deploying it in an enterprise for ~5k users, and it works surprisingly well for providing the ability to run arbitrary commands on instances without ssh access.
[1] https://docs.aws.amazon.com/systems-manager/latest/userguide...
Does this refer to an SSH password or e.g. a 4096-bit RSA key, or possibly something completely different? If this refers to an RSA key then I'm quite confused.
If you have a secret, it's either (potentially) compromised and should be changed immediately, or it's not compromised and thus doesn't need to be changed.
If you're not sure if it's compromised or not, you have an entirely different problem that isn't solved by rotating. Put another way, rotating every 90 days is the equivalent of saying "it's okay if this secret is compromised, because no one can do any damage with it if they only have under 90 days" -- and I can't imagine any situation in which that would be true or acceptable.
But in another sense, all your secrets are potentially compromised, all the time. That is, there are compromises that you don't detect, or don't detect right away. They happen. What should you do? Well, you change the secrets every 90 days, even if you don't (yet) know that they have been compromised, to minimize the damage of a compromise that you don't know about.
But, you may say, by that logic shouldn't we change them every day? Every hour? But the other tradeoff is with annoying your users, wasting peoples' time on changing secrets, wasting time as they have to try to remember today's secret, and all the associated hassle. It's a compromise between better security and better usability.
Is 90 days the right compromise? I don't know.
* Do not impose other composition rules (e.g. mixtures of different character types) on memorized secrets.
* Do not require that memorized secrets be changed arbitrarily (e.g., periodically) unless there is a user request or evidence of authenticator compromise. (See Section 5.1.1 for additional information).
There's research [2] backing the idea that these policies actually lower security because of the way users act in response.
Systems don't have these problems, so rotating every few seconds doesn't seem to be a big deal. However, if the secret was compromised and you don't know how, you have to assume that the new secret can also be compromised (via the same mechanism), and at the same time there's no reason to assume that an attacker can't make use of it before the next rotation. Thus the rotation is completely ineffective as a means against this unknown attack, so why bother? The energy would be better spent eliminating potential for unknown, undetected attacks.
[1] https://pages.nist.gov/800-63-3/sp800-63b.html#-1021-memoriz...
[2] https://www.ftc.gov/news-events/blogs/techftc/2016/03/time-r...
Think both/and, not either/or. Yes, work on eliminating unknown, undetected attacks. But you're never going to be perfect at it, so also work on minimizing the damage from undetected attacks. If you're not dealing with human-remembered secrets, sure, go down to rotating every few seconds. By doing so, you make the compromise harder to exploit. Even though you don't make it impossible to exploit, "harder" is still worth doing.
This attitude is why company after company keeps leaking customer data.
[secret rotation,] that's not something that most of the time makes sense to use.
Secrets Manager is very likely backed by KMS which appears to use a KMS-specific HSM module (judging by https://csrc.nist.gov/csrc/media/projects/cryptographic-modu...)
Because it's free they limit the requests per seconds you can make to the Parameter Store. That's especially noticeable when doing requests for all parameters of a given path, as the limit is way lower there than for requesting (a bunch of) individual parameters. In the past that caused serious problems for us when using Parameter Store for AWS Lambda functions during a deploy of new versions of functions, as suddenly there was a spike in the number of requests to Parameter Store as all AWS Lambda containers got replaced.
They of course set such limits because it's free, so I'd gladly pay for getting increased limits.
Would have been nice if there was a 5 secret free tier though. I feel like the sticker shock will dissuade small users when everybody really should be doing secret rotation.
It can either write the environment variables out to a Docker compatible environment file or it can be used as a wrapper to run any executable/script with the environment pre-populated. The README contains various examples of those use cases.
The primary motivation was to have proper secrets management for my hobby projects that didn't rely on Ansible Vault, configuration files in S3 or having to use the AWS SDK in each project.
If you are using IAM roles, a HTTP call to http://169.254.169.254/latest/meta-data/iam/security-credent... would give any attacker with shell access the credentials needed to iterate through any secrets in the Parameter Store that the instance has been given access to.
For shell access as another (non-root) user that is not running your application, it is just as safe to rely on environment variables as having configuration files with proper file permissions: https://security.stackexchange.com/questions/14000/environme...
The killer-app of odin imho was/is the on-machine http server that let all manner of applications very easily get credentials.
You could just do something like
MY_API_KEY="$(curl http://localhost:5000?key=my-api)"
The fact that Amazon systems bootstrapped EC2 instances meant they could easily enable this org-wide. The odin back-end then owned all the lifecycles around those secrets including what kinds of hosts they would go to, if/how they would rotate, etc etc. It has its annoyances including the fact you have to use http to localhost which can get saturated if you're not doing things right, but overall it really made secrets-management a non-issue.
Suppose you have a service that fetches snippets from user pasted links (like Slack does). All I have to do is paste a link of "http://localhost:5000?key=my-api" and your server would return a snippet containing the secret.
Thats why Google Cloud and others require a special header to be set. Hopefully Odin does the same? https://cloud.google.com/compute/docs/storing-retrieving-met...
https://ionize.com.au/stealing-amazon-ec2-keys-via-xss-vulne...
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-role...
TL;DR;
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access
Brazil's dependency model is, I think, very much worth rethinking. Just try to use it for NodeJS. Apollo is fine but slow and dated and needs love. Pipelines works for specific software models and not others (ever seen a pipeline with 500 envs?). Great design for building custom approval steps though (long ago I wrote a now-popular one).
Our team was abandoning Apollo for MAWS, so our deployments were bridged between the two, resulting in another twisted pile of incomprehensible script-glue.
Pipelines was ok (!)
Everything on public EC2 for managing deployments (CodeDeploy and CloudFormation in particular) were so broken and bug-ridden and non-performant that we couldn't reliably deploy production code with them.
I prefer open source solutions in almost every way. I actually can't think of a single problem that the Amazon internal stack solves that isn't better solved in the open-source world. Which isn't surprising, given the constraints involved.
You do still have to figure out a way to securely provide the root credential to your service so that it can fetch the secrets from the secret manager, correct? Otherwise this would be magic of a kind I think is impossible.
If my questions aren't too far off in the weeds, then this service sounds like a personal password manager but for a service rather than a person, though I'm sure AWS's service has finer-grained controls than just the all-or-nothing master passphrase. Similar risks apply: an attacker obtaining the master passphrase is a major issue, losing the master passphrase is devastating (though recoverable here because you probably didn't lose your personal AWS login credentials), and unavailability of the password database is catastrophic. But the usability benefits of having everything in one secure place, behind a service managed by experts, should outweigh those risks.
I have more questions about the credential-rotation feature, but this is enough for now.
Access keys can be provisioned and downloaded straight onto the box from the service. Sure, a compromise is bad, but only exposes the secrets that would be available on the pwned box regardless.
So in other words – a unique key is generated every time a virtual machine is created. It's fully automated, never shared between instances, and never needs to be handled manually. That key will give the virtual machine permission to access other AWS services, in this case the AWS Secrets Manager.
So as long as you're using EC2 instances, you won't need to worry about securely passing a 'master password' to your VMs in order for them to access secrets.
I exist, therefore I can.
Having your personal AWS keys compromised should also be a non-issue because of MFA. My keys that I keep in OSX Keychain grant zero permissions, except the ability to assume a more privileged role, but that requires MFA.
FWIW, we're using Parameter Store for secrets and it works great. Given that I just finished that set up just weeks ago, I'm in no rush to jump on the Secrets Manager wagon based on what I'm seeing.
The only problem with both services is the 4k character limit. We have some big cert chains and such that exceed that, so we had to build the tools such that they could put/get a chunked secret. Wasn't too bad though.
There is a point where Vault is more cost effective, but I believe it'd require a ton of requests and secrets to justify min 6 machines of at least t2.micro that also need to managed and secured.
You can also back Vault with something other than Consul. You can back it with DynamoDB, which would be much cheaper than managing your own Consul cluster. You can even back it with S3, which would be dirt cheap (cost of the vault instance + a few cents for storage).
With Vault 1) you have to get Hashi Enterprise 2) Pay for a very pricey HSM.
A SecureString can be `postgres://admin:hunter2@localhost:5432/db`. It can also be `{"username": "admin", "password": "hunter2", "host": "localhost", "port": "5432"}`.
WOWZER. I get having a managed solution is great, but you don't have to store many secrets before running your own Vault server makes sense.
That being said, if the security traffic and/or amount of data is huge, Vault might be cheaper, even with the extra work. I guess it depends on the scenario.
The real issue for some of the people at my work is the vendor lock-in versus vault.
Point being: if you're using a modern stack, you can use modern tools. If you're stuck on a legacy stack, you use legacy tools, and that appears to be who AWS is building for nowadays. Legacy tools, value-based pricing instead of cost-based, charge a boatload, buzzword heaven, sell it to the enterprise, rinse repeat.
The HSM component is the only part here that appears to be worth the price. Azure's competing product is a LOT cheaper, except that their HSM-generated keys are more expensive. Which naturally means, Azure is cost-based pricing, whereas AWS is value-based pricing.
These were mentioned around the custom rotation strategy, any pointers to docs describing how it is done if the secret is an RDS Secret?
I used to think that Cloudwatch metrics were very expensive at $.50 per custom metric per month, but this seems waaayyyy cheaper to store.
AWS Secrets Manager is for storing credentials to products (e.g. DBs) and have processes for automatically rotating them on a regular basis.
KMS provides an alternative to an HSM in the cloud where this service is alternative to running a Vault cluster with a backing Consul cluster.
Looks like a cool product I noticed they dont include a revoke workflow though, the trickiest bit of key rotation
1. Use a KMS key to encrypt a secrets file (obviously, never check this into source control)
2. Store the encrypted secrets file in an S3 bucket
3. Tie a new IAM role with kms:Decrypt and s3:GetObject policies for the relevant resources to your EC2 instance
4. On app start, get the KMS key and secrets file, decrypt, and set environment variables
In practice, rotating using this scheme just means creating a new KMS key, re-encrypting the file and pushing the updated copy to S3, and updating the IAM role's kms:Decrypt policy. It's not too bad unless you have a million services.
KMS keys on the other hand, do not have that use case. If the secrets in your S3 bucket have been encrypted using a KMS key (or better yet, a data key derived from a KMS key) and your bucket is compromised, your secrets are safe as long as the attacker cannot ALSO get access to the KMS key. By all means, enable encryption at rest on S3, but don't make that your only defense.
That would get smaller users to start using it instead of parameter store and eventually realize the value of automated and audited secret rotation
If so, thats pretty damn cool.
It can provision users to backends like SSH, databases, cloud providers, and such. Use is audited, can be revoked, and has a TTL associated.
Additionally, vault contains a full "crypto-in-a-box" implementation that allows for sign/verify, hmac/verify, encrypt/decrypt, random number generation, and other functions.
So I applaud AWS for doing this and hope the will continue developing KMS/HSM/Parameter Store/Secret Store/??? in the future and innovating, but evaluating Secret Store vs. Vault simply on price may be a short sighted comparison.
Disclaimers: Employer is an AWS customer using vault
Unfortunately — although I'm sure it's built on-top of ParameterStore internally — I just checked and you can't see SecretsManager secrets in ParameterStore, so an application would need to read from both and merge them, or switch entirely to SecretsManager to take advantage of the automatic rotation.
But not with this price per secret...
However, the biggest delta is: I'm like 90% sure Azure key vault doesn't have fine-grained access policy per VAULT. That kind of stinks.. you need a vault per role ideally.