Mozilla SOPS with KMS and Git is underrated (2019)

I'm one of the creators of Vault. I read this back when it was posted and I'd be happy to share my thoughts. I'll note its worth reading through to the last paragraph and into the comments, the title is a bit bait-y and the article does a better job than the title gives itself credit for.

Broadly speaking, if you're looking at Vault to solve a specific problem X for a specific consumption type Y on a specific platform Z, then it _probably is overkill_ (I wouldn't say "overhyped" :)). i.e. "encrypted key/value via env vars on AWS Lambda". "X via Y on Z." The power of Vault is: multiple use cases, multiple consumption modes, and multiple platforms supported with a single consistent way to do access control, audit logging, operations, etc.

I can't stress that "single consistent way to do access control, audit logging, operations, etc." enough. Multiple security use cases dangling off that consistency is really important as soon as you hit N=2 or N=3 security use cases.

If you need say... encrypted KV and encryption-as-a-service and dynamic just-in-time credentials and a PKI system (certificate), and you need this as files and as env vars, and you need this on Kubernetes and maybe also on EC2, then Vault is -- in my totally biased opinion -- going to blow any other option out of the water.

That's a somewhat complex use case but its something Vault excels at. For simpler use cases, Vault is making more and more sense as we continue to make Vault easier to use. For example, we now provide a Helm chart and official K8S integration so you can run Vault on K8S very easily. And in this mode, developers don't even need to know Vault is there cause their secrets show up as env vars and files just like normal K8S secrets would.

Also, this article is from June 2019 and in 10 short months we've made a ton of progress on simplifying Vault so it gets closer to that "X via Y on Z" use case. Here are some highlights I can think of off the top of my head but there are definitely more, this is just from memory:

* We have integrated storage as an option now, so you don't need separate storage mechanisms.

* Our learn guides went from basically zero to lots of content which makes it much easier to learn how to use Vault: https://learn.hashicorp.com/vault

* We have an official, feature-packed Kubernetes integration to do stuff like secret injection and rotation automatically. We also publish a Helm chart to run Vault on Kubernetes. https://learn.hashicorp.com/vault?track=getting-started-k8s#...

We're looking at ways to make running Vault much, much easier. More on that later this year. :)

If you're thinking of HashiCorp Vault as a platform that is solely there for secure storage of static secrets (i.e. basically a secure key/value store), then you probably would think it is over-hyped.

The power of Vault is in the integrations and the unification of authorization policies across those different integrations.

"I need a way to allow my different AWS accounts to issue TLS certificates within particular namespaces of a common CA" - Vault can do that.

"I am managing service identity using mTLS at the moment, but now I need to get short-lived SSH keys for those service identities" - Vault can do that too.

"I need to issue credentials that allow access to AWS and GCP IAM roles to my on-premise application which has a JWT" - you guessed it, Vault can do that.

Sops creator and co-maintainer here. We love Sops. We love Vault. We use both of them at Mozilla in various places. Beyond the "tool war", which is silly, what really matters is using tools that integrate well with your workflow, provide real value and strong security.

Not to overreact, but this seems like a trashy hit piece.

Vault checks all the boxes for the authors own “Ideal Secrets Management Solution Requirements“.

If Vault’s only feature was its PKI, it would be underhyped in my opinion.

This is a bad analysis. Sure, Vault is infrastructure you have to maintain. For a small team, it's probably not a good tradeoff. But even if you dismiss Vault's ability to do dynamic secrets (which is actually one of its primary benefits), the security model for Vault is much stronger than KMS-encrypted secrets in Git. The author appears to be claiming that protecting your Vault server from a local root privilege escalation attack is so difficult that it's better to ... instead rely on the security of your team members' IAM credentials which are floating around on their workstations, and which talk to KMS over a public API.

No matter how you manage your secrets, one of the most important factors is how often you rotate your secrets. If you can't transition to fully dynamic just-in-time secret provisioning that Vault can provide for certain backends (including AD, OpenLDAP, PKI certs, various database engines, Consul, and more), then you still ought to be rotating your secrets on a regular basis. And in that case, a tool like Vault (or AWS Secrets Manager or GCP Secret Manager) is going to be a lot easier to build a rotation framework on top of than a git repo with secrets encrypted in place.

If you have a handful (say less than 10) repos, the article is right – Vault is an overkill. You can and should just check-in encrypted secrets with git-crypt or SOPS. Whenever master enc/dec keys are rotated, you can afford to make a dozen PRs and commits. Note here - I'm not talking about rotating revoked secrets. I'm taking about the master key rotation.

If you have hundreds of repos, the mere fact of "rotating" all secrets when the master key changes is a laborious task. Opening, tracking and waiting for hundreds of PRs to be merged, with stuff failing for unrelated reasons to secrets (random unit test or PR check failures, etc) will take months.

https://12factor.net/config – this exists for a reason. Your PR checks have nothing to do with production secrets. Your code is compiled with Java/Go/Node/whatever and is published as an artifact, and this also has nothing correlated to secrets nor configuration. You should be able to run the same artifact (docker image or jar or whatever) – with different configurations and secrets for different environments.

While the article is good and technical, it skips over these parts. If you're just doing a toy service with no regard for anything of the above, then yeah, by all means just use SOPS and don't bother with vault. But if you're an organization, then don't read this assume that this works for 100s of services & engineers with differing CI/CD practices and schedules, and no luxury time to wait hours for a key=value change or waiting months for a full rotation.

Vault power comes in its rich integration.

Exampme, it can authenticate using K8S JWT, or AWS IAM. Using K8S JWT, the process of generation token is gone, you use the token from k8s. Send it to Vault.

If you just look for something like a key-value store for env var and authorized using simple username/password pair then KMS or any key-value store with a HTTP on top of it will work.

Anyone here used Vault to secure a Phoenix application? In particular, the idea that a short lived database credential can get auto-refreshed and then notify the Repo's supervisor to restart the repo with new credentials?

That's how I'd probably do it but I'd rather not reinvent the wheel if someone's already done it.

Until you dig into deeper use cases, it's easy to get hung up on Vault as JUST a KV store. Honestly, if that's all you want, and you're on a cloud provider, using a combo of their storage + KMS or other services (i.e. SSM Parameter Store) is probably the more practical approach.

That said, the proposed drawbacks aren't necessarily ... drawbacks? Granted my perspective is from deploying Vault to the Cloud vs. on-prem.

1. Yeah, Vault needs a separate place to do its storage. However, that's a strength if you're following an immutable infrastructure pattern. You store the data in something like DynamoDB and still have the freedom of tearing down and re-creating the Vault servers themselves.

2. Vault may be more expensive right out of the gate, but if you're trying to cover ALL of its functionality with cloud services, it'll start saving you money eventually. Furthermore, many of the cloud alternatives have service limits and quotas. I mean geez, if you want an internal CA through AWS, you're paying a flat $400 a month + costs per certificate.

3. Vault has a learning curve, but it's not worse than having to memorize the buffet of CLI commands through your cloud provider. Yes, getting it set up for the first time can be a jigsaw puzzle, but when everything is up and running, it's smooth sailing. (Plug - I have a project that automates setting up Vault on AWS: https://github.com/jcolemorrison/vault-on-aws)

4. As for vulnerabilities of the "default implementation" - Yes, the public cloud presents more opportunities for exposure, but that's not limited to just Vault. Furthermore, if someone gets root access to your vault servers...that's not a vault thing. 80% of the 2019 massive cloud breaches are the result of misconfigurations and account compromises (source: https://www.paloaltonetworks.com/resources/research/unit42-c...).

I cannot comment on Vault as I have not tried it yet, but I am a happy user of sops for some deployments.

In my case the secrets are stored in a sops-encrypted JSON file in the repository. Sops encrypts the secrets but keeps the JSON structure so changes to a specific secret can be tracked in git with a oneline change instead of a binary blob diff of the full file which is very nice.

10 months ago, 62 days ago and then again today. You should find something else to share instead of vault. https://news.ycombinator.com/from?site=oteemo.com

Note- I dont work for Hashicorp but have used many of their excellent products from time to time.