Hacker News Reader

Comment by jt2190 | Hacker News Reader

The unwritten assumption in 12 Factor is: the environment is secure. For example, a production system should always have a secure means of setting environment variables. Said another way: If a random dev can change an environment variable in production either directly by logging in or indirectly by pushing code then there is something very very wrong.

If the dev is pushing code to production they should not simultaneously be pushing environment configs, this is doing two logically distinct things at once: Changing application behavior AND reconfiguring the server environment.

If the dev is adding secrets to their local config and they’re pushing that config to insecure places that means their deployment pipeline is broken and it should be fixed. .env is never committed to source for this reason, for example.

View on HN

> The unwritten assumption in 12 Factor is: the environment is secure.

Fair assumption.

Assuming no attackers and you're only running trusted code, I still maintain the environment is a poor place to keep secrets. Devs adding `{ meta: process.env }` to logs. Instrumentation/reporting libraries dumping the process (and the env) for crash reports. Trust that subprocesses + dependencies inheriting your environment are taking equal care to avoid these issues, too.

Unfortunately, with secrets in the OS env, you‘re one `printenv` or improperly written third party dependency that leaks env vars away from a security incident.

The env and more importantly what populates it should be secure, but security works best in layers. Sanitizing the env after loading it is a nicer middleground, k8s-style secrets materialized to files work best and are conceptually close enough to the OS env.

It's not my intuition that materializing secrets to files is a better way to protect them than just injecting them into the environment, where they don't persist.

The env is technically still kind of a file on linux at least (through /proc).

Sometimes I feel like stdin or an unlinked memory mapped file might be the best location for this stuff. Wish Linux had a cloexec+1 option, where an fd is closed after two execs, so you can set up a child process for success.

Only in the sense that everything is technically a kind of a file given access to proc.

Dev/shm is used to materialize them, and then you have the ability to isolate the downstream code you might use from accessing it by dropping permissions or sandboxing it away from a file. You cannot really hide your environment from anything in process, since it's such a low level construct.

Thus it's easier to leak environment unintentionally, leaking file contents takes effort.

Not at all clear why this is a better setup than a launcher shim that pulls secrets from a secret store and injects them into the environment as the program launches --- which is a pretty normal shape for these things to take.

I guess you could be thinking "subprocess inheritance" as a downside? But subprocesses often need secrets, and if you arrange for that with the filesystem you have the same problem. And, of course, files leak all the time.

More to the point, though: none of this has anything to do with whether you should add secrets to your .bashrc or whatever, which is the argument I'm seeing on the thread.

Have you seen this where eBPF patches secrets inside TLS send buffers? https://github.com/spinningfactory/kloak

Now you have to be much more clever to leak them :)

(Edit: I need to read up on this better)

It doesn't need to be a traditional file. You can pass it as essentially a read-once file by using stdin. Depending on your desire for modernity, similar behavior can be obtained by leaving a file handle open for the exec-ed process to inherit, via a Unix socket, or even a lightweight TCP daemon.

> It's not my intuition that materializing secrets to files is a better way to protect them than just injecting them into the environment, where they don't persist.

I don't really follow this reasoning. Where are you injecting the secrets from?

> you‘re one `printenv` or improperly written third party dependency that leaks env vars away from a security incident.

Game over already if anyone can run commands or arbitrary code. Not using the environment won't help you.

This was about a programmer accidentally adding a debug print, or say an error page helpfully dumping environment, or logs from a third party tool, etc. Not about someone actually getting RCE just to print the environment, of course.

systemd-creds too, for anyone who has not found it yet like I was a week ago.

tbf thats what a good logging lib is for, and if printenv can be run within the machine, then it's already too late

Adding on to what others say about printenv, various diagnostic tools (e.g. crash reporting stuff) will capture the environment. Env vars are just categorically so easy to accidentally leak that it can’t even be classed as an insecurity.

> Changing application behavior

Configuration changes also change application behavior, otherwise is it really “config”?

> Note that this definition of “config” does not include internal application config, such as config/routes.rb in Rails, or how code modules are connected in Spring. This type of config does not vary between deploys, and so is best done in the code.

I guess it depends on your definition of "behavior?" For example if the config is the endpoint address of an external resource, it's not really changing the application behavior per se.

Adding a config setting should never be dangerous (if it is your system is deeply broken) and should be distinct from changing an existing config setting.

> Adding a config setting should never be dangerous (if it is your system is deeply broken)

While I can't name anything specific offhand, I feel pretty strongly that I've seen documentation for various things stating that those things check for an environment variable and, if it isn't present, fall back to other candidate names for the same variable.

This makes setting a new variable synonymous with changing an existing one, unless all variables are currently using the highest-priority possible names.

Another architecture with the same effect is that the software will only check a single environment variable, and if not present it will use a default value. That also makes setting a new variable synonymous with changing an existing one.

Never seen this problem in the wild. Seems like it would be a very rare issue limited to very large configs with conflicting names.

You've never seen software that will use a default value, instead of refusing to operate, when a particular environment variable isn't present in the environment?

No I’ve never seen software that broke because someone added a config key.

Are you talking about altering environment variables in a system, or altering software to read different environment variables? I read jt2190 as talking about the former.

I don’t use env variables for this and many other reasons. They seem like a spectacularly bad idea for config, as the op noted.

I’m talking about altering a config which you control - adding to a config is pretty risk free if it is your program, altering a config of course is not.

I was deploying some dotnet app and it broke because the devs baked a config key into the image that was otherwise unset, which enabled it trying to start a SSL endpoint without a cert, thus breaking the app.

The assumption is the vector. Why assume?

It's more accurate to describe it as a "premise", not an assumption. It may not be true of every environment, but it's a very common norm (for instance, secrets management systems inject tokens and such through the environment).

You can reject the premise in your own environments, and then that part of 12 Factor doesn't apply to you.

Reply on news.ycombinator.com