Hacker News Reader

Comment by arkadiyt | Hacker News Reader

The author touched on it a bit but I personally don't see the benefit of using Secrets Manager over Parameter Store:

- Parameter Store works just as well and is free (outside of the KMS key cost)

- Tools like Chamber only support Parameter Store

- The one benefit that SM had for me was out of the box support for rotating RDS passwords, but RDS authentication with an IAM role is an even better solution for that

View on HN

I've been bitten by Parameter Store's rate limiting (non-increasable) from just launching a new environment of a regular application (we were storing a lot of values, and there's no real bulk GET as far as I recall). There are countless threads about this problem online.

I imagine that with a serveless workflow, this might be an even bigger problem.

I haven't explored Secrets Manager yet though.

Fetching parameters by path is an easy way to get bulk parameters and avoiding the rate limits.

It has caused us failures as lambdas have scaled up as it is quite easy to hit a rate limit.

For me, I was able to reduce the calls from lambda pretty significantly by doing the parameter pull step outside of the handler, so it is available as part of the execution context between runs (similar to how many people do DB connection with lambda). I still do the decrypt step within the handler though because I don't like the idea of having decrypted values lying around, but the limit for KMS decrypt operations is independent of SSM, and is a 10k/s shared limit in the regions I use, so I haven't had any issues yet.

I’ve been bitten by it doing something as simple as creating parameters using CloudFormation. The only way around it is judicious use of DependsOn to make them single threaded.

Thanks for this, I was considering bringing params into CF, the main blocker being that it doesn't support secure params (which is obvious in retrospect, they wouldn't be secure in the template).

For secure strings in cloud formation, I use a cloud formation parameter ( - meaning I have to enter the parameter manually and set NoEcho to true. And then set the CF parameter value to the SSM Parameter value.

This is with the help of a lambda backed custom resource

https://svdgraaf.nl/2018/04/13/CloudFormation-ssm-secure-str...

Obviously , this can’t be a part of your CI/CD Pipeline. I run the CloudFormation template manually in the console to enter the parameters and my CI Pipeline can then update the stack when needed.

SSM parameter store only stores one thing at a time, but SM stores things in key/value pairs which you can retrieve all at once.

Make your value a JSON string, problem solved ¯\_(ツ)_/¯

You can definitely retrieve multiple values, stored as k/v pairs at once with SSM parameter store.

https://docs.aws.amazon.com/cli/latest/reference/ssm/get-par...

Just use the --recursive option.

I'm of the opinion that AWS just doesn't like people using SSM Parameter store w/ KMS encryption as the main solution b/c they realized too late that they missed an opportunity to charge money for it.

I’d rather they charge for it. They need some incentive to keep developing it, and parameter store is frankly atrocious.

If you're using path prefixes you can retrieve all of them easily using the AWS SDK[1,2].

One of the things that we do is to prefix all of our SSM keys with a prefix based on the service name (for example, `example-api/`) and fetch them by path rather than individually.

[1] https://docs.aws.amazon.com/sdk-for-go/api/service/ssm/#SSM....

[2] https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SSM....

Yeah, the prefixes are killer too because they work well so with IAM. So you if you have multiple teams sharing an AWS account, you can also prefix by team name and limit decryption rights to those users and their resources.

The only downside is that the path is limited to 5 items IIRC. Which you can actually get to pretty fast with stuff like:

/team/service/environment/keyParent/keyChild

"environment" above would be like "prod" or "dev".

I didn't know about the limit although you might be able to get it bumped?

In my experience we haven't had any issues getting to that limit but we only ever use 1 level of depth (the service) and each environment has it's own AWS account so there is no path for environment.

Each service has a IAM Role with an IAM Policy attached that gives it access to all of the keys under the service path.

A way to fix the team prefix (what if 2 teams actually need access to that secret?) would be to not have it in the path but use roles that teams can assume to get access to those secrets.

Yeah, I personally find using separate AWS accounts for dev/test/stage to be a bit of a PITA overall, prefer to isolate those resources via VPCs & IAM policies in an umbrella non-prod account, but definitely see how that would simplify other aspects of setup.

The path limit isn't a huge issue, just something you have to design around up front. As you mentioned in another comment, no reason you can't store a JSON or something in the string if you need deeper nesting.

For secrets needing to be shared between teams I'd just create a different prefix in the place of team, like /shared/, and grant access appropriately. Or just use no team prefix at all and make it accessible if it's for everyone.

But it's quite nice that the service is flexible enough to be useful for a variety of use cases / infrastructure setups.

VPCs are not a substitute for accounts. Most AWS resources must have unique names across the account like lambdas, SQS queues, sns topics etc.

When you have one account you end up prefixing/suffixing resource names for different environments. It gets to be really ugly.

Not a problem. Just deploy with CloudFormation and it will give you unique names for your resources automatically. You can use the same template for everything and filter by the stack tags it auto adds. Still way easier than dealing with creds, billing, and monitoring for 3 different non-prod accounts. Though I admit that the new aws organizations feature has made the billing setup stuff a little easier.

it also gets pricey if you have a lot of secrets. $0.40/secret

Reply on news.ycombinator.com