Hacker News Reader

Comment by gchamonlive | Hacker News Reader

I am progressively adopting Hashicorp Vault as the secrets manager of choice. It can be used by a variety of different scenarios -- directly into the application using AppRole, with Terraform using it's secrets provider, by developers, during vault authentication when they get their secrets and access regenerated.

This way I am not bound to docker swarm, or keywhiz, or god forbid AWS Secrets Manager.

As of now, I am still exposing secrets with Env Vars, but the next step is to use Vault directly. Vault has been pretty reliable so far. It is using AWS KMS for managing the master key and a scalable DynamoDB table for high availability backend.

View on HN

Vault looks cool, but looking at the reference architecture [1] my guts tell me it's much easier to fuck up setting up a Vault cluster than environment variables.

[1] https://learn.hashicorp.com/vault/operations/ops-reference-a...

You can always use their official Terraform module for AWS if you are not comfortable with setting it all up by yourself: https://registry.terraform.io/modules/hashicorp/vault/aws/0....

But if you cut Consul for backend, if you are not using consul for other service discovery and Nomad etc..., you can simplify that deployment a whole lot. You make sure you open the cluster communication port, setup a Application Load Balancer in front of the cluster to balance traffic and serve SSL, configure auto-unseal using AWS KMS (since you are not using it too often, 1$ is OK to have AWS manage your master key), deploy Vault on every cluster instance, and use something managed like DynamoDB as your backend. I think this is a pretty simple yet scalable setup. The amount of cloud lock-in is pretty minimal and can easily be replaced with HAProxy setup.

If you are in AWS, you can use S3 for data storage and DynamoDB for HA. Our first install was set up using their 'best practices' and consul, being stateful, is frustrating to run in kubernetes. We migrated to using S3/Dynamo and now have fewer moving parts and haven't seen any issues.

Thanks for the advice!

What's wrong with AWS Secrets Manager?

If I'm already working 100% in Amazon, I'm tempted to use Secrets Manager rather than justify the cost in hours to deploy and maintain a Vault cluster.

Interested in your opinion.

Cloud lock-in (not being able for example to manage cross-platform credentials) and cost.

By itself AWS Secrets Manager is fine, but as with everything AWS, billing is pretty opaque. If you couple this with a poorly planed Lambda setup, costs can potentially uncontrollably scale.

For what you are paying, AWS Secrets Manager seems too barebones/rigid.

Lock in shouldn't be too bad. It's create, get, set, list, delete, describe. I imagine the API footprint isn't much different if you switch over to something else. Most apps will just use get().

It shouldn't, but it can be very bad. And with vault you potentially circumvent that and end up with a tool that is way more flexible than Secrets Manager.

Not OP, but I've recently worked with AWS Parameter Store a lot, and I've been very happy with it so far. We store all configuration (including secrets) using it.

We started by using https://github.com/segmentio/chamber/, but because of the way we decided to structure our secrets we decided to write a clone of it ourselves (https://github.com/micvbang/confman-go).

In practice, we have a client written in Python that grabs configuration from Parameter Store and puts it into the process' ENV variables at startup. This allows us to avoid vendor lockin in the rest of our code, since we still just fetch all config from the environment. I like it so far :)

IMO, it gets expensive quickly, and it was designed around AWS' use case, which involves credentials that roll on a regular basis. Outside of RDS, its value for the price goes downhill quickly.

To a sister comment who mentions parameter store - that UI is the biggest leaking bag of horse manure I've ever had the displeasure of using. We made the mistake of using it, and moved over to Vault at the first available opportunity. Vault isn't a silver bullet either, but the UI is at least usable.

I assume you mean Parameter Store's web console UI? I've never really used it; we use only API endpoints and command-line tools, and it works great.

The web UI is bad, but tools like chamber (https://github.com/segmentio/chamber) are excellent, so there's no need to deal with the web UI.

> This way I am not bound to [...] keywhiz

No, but you’re bound to Vault?

I agree that Vault is likely the best solution to this (dynamic secrets are super useful), but “not being coupled to something” is not a benefit you get from it.

You are right, I think that was a little harsh on my part. I can't really say much about keywhiz, I should have left it out of my comment.

Yes, you are tied to Hashicorp Vault once you adopt it, but it isn't much of a problem because you can easily redeploy it in a different cloud.

It might be true to keywhiz, so my comment wouldn't be valid to this point.

How do you share the token to access Vault to your code?

Using AppRoles and consul-template. And if you're on Kubernetes, you can use https://github.com/sethvargo/vault-kubernetes-authenticator in an init container.

And how does k8s pass that value to the app?

You can use consul-template to generate a config file that lives in-memory (volumes.myvolume.emptyDir.medium = InMemory in a PodSpec).

Reply on news.ycombinator.com