I've been running K8S for the past 6 months. So far, so good. Helm helped us a lot in our workflow. A lot of trial and errors, but to answer your points, this is what we're doing so far:
> * Where do you store the YAML manifests, and how do you maintain them? Do you put them in the same git repo as your app, and if so, how do you deal with the fact that the YAML files are going to be different for production, staging, testing/QA, etc.? (For example, ingresses will use other host names. Configs are likely to be rather different altogether.)
We store the manifests in the same git repo as the app. Docker images and YAML files are all the same, what's different are the values, so we use helm and override the values per environment.
The values are split 4 ways:
- default sensible values (values.yaml). For example, feature flags.
- datacenter specific (gcp.yaml, aws.yaml). For example, REDIS_HOST and MYSQL_HOST.
- environment values (env-production.yaml or env-staging.yaml). For example, the ingress values and its certs, how many replicas for each service
- secrets.yaml (not stored in git, but generated on the fly, will explain more later). For example, MYSQL_PASSWORD
I mean, if you're doing your services right, they should all have the same docker images, only the config changes. Helm has a pretty good templating and config helpers.
> * Or if you centralize them in a single git repo, you have to make sure that you always pull the newest version, and that your workflow includes diffing against the currently deployed version, and so on.
Helm takes care of this. When you upgrade a deployment, it looks like it's doing a diff. What's cool though, is that it seems like it does a diff on everything except replicas, which is exactly what we needed.
> * How do you protect configs/secrets, if they're in git and available to all?
I don't mind having the config in git, I do when it comes to secret. We store secrets as a gitlab variable, and our CI dynamically creates a "secrets.yaml" before deploying. I'm still not really happy about this though, I think a better way would be to use Vault, but it adds some complexities that I'm not really to deal with yet.
> * Dependency tracking: If app A needs app B, you want to encapsulate that dependency in your workflow.
I'm not too sure I understand what you mean by this one. We, thankfully, merged all our git repos into 1 monolithic repository. I remember at first reading about the big boys (FB, MS, Google) having a single repository and thought they were crazy. Then we added more and more services and suddenly, I just realized that having a monorepo makes is a lot easier for developers and ops. AppA~1.0 needs AppB~2.5. We used to have a crazy graph of dependency like that. Not anymore. Now every services has the same version -- we're simply using git hashes. AppA~4c20d needs AppB-4c20d. Every push to the repo rebuilds the docker images and tags them like this: AppA:master-4c20d and AppB:master-4c20d. And yes, we build for all branches and all commits.
So for deploying, it's super easy. I just deploy everything at once, at all time. If I need to rollback, I can rollback to a specific version on all services. If I need to test in a real environment a specific version, all I do is add --set BRANCH=fix-bug-branch,COMMIT=4c20d
> * Continuous delivery: How do you take care of these concerns in relation to the CD system (e.g. Drone, Jenkins)?
There's no concerns here since every code push starts our pipeline, and the pipeline rebuilds all docker images, and runs the tests between each other. It's pretty dope.
> * And of course, you'll want to be able to develop locally. If you run Minikube, how do interact with it, YAML-wise?
We have a local bare-metal server running a single-node k8s, but to be honest, we've never had a dev needing to develop on K8S specifically. Each dev that builds a service is also tasked with building the dockerfile that goes with it, and with assistance, they build the YAML file that goes with it. It's pretty straightforward. Our gitlab auto deploys to dev, staging and review environment, so if there's an invalid YAML, the pipeline will catch it.
> There's Helm, but Helm is essentially just a "templates in a central repo" manager. It doesn't solve the configuration issue: You still need to provide values to the charts.
I think it does but not directly. It solves it because it allows you to use templates and override config in a sane way.
Here's an extract of our .gitlab-ci file for deploying to production
- echo "Deploying to Kubernetes Env=production Branch=$CI_BUILD_REF_NAME Version=$CI_BUILD_REF"
- "echo MYSQL_PASSWORD: $MYSQL_PASSWORD_PRODUCTION > tmp.yaml"
- "echo MYSQL_USERNAME: $MYSQL_USERNAME_PRODUCTION >> tmp.yaml"
- ... more secrets echo config here
- helm upgrade master ./manifests/App -f ./manifests/gcp.yaml -f ./manifests/env-production.yaml -f tmp.yaml --set BRANCH=master,COMMIT=$CI_BUILD_REF
> Right now I'm debating whether to just go for "simple and stupid" and have a central repo, and then build a small toolset to wrap kubectl for devs [...]
We started with building our own toolset to wrap kubectl. It worked. Then we found Helm. I remember at first looking at it and thought it wasn't useful for us.. now it looks pretty good. We heavily use templates to reduce boilerplate.
-----
I'm curious about your tool called Monkey. Is it a frontend for chef or ansible? Also, I can pretty much ask all the same questions you've asked but geared towards your tool (where do you store manifests, how do you protect config/secrets, how do do you handle dependency, continuous delivery, etc etc).