I wonder if anyone else does this.
I have a private, env-var only repo per project that I very simply import (git clone) while building. I'm sure there are some risks like possible leakage if a team member's github account is compromised but it seems like a very nice, clean way of managing secrets. I even get the benefit of a version control.