Sucks that there isn’t anything simpler than k8s that’s production grade.
Maybe it’s time to yolo with a regular container that just restarts on failures, ha…
Maybe it’s time to yolo with a regular container that just restarts on failures, ha…
It's a little surprising how many folks are unaware of the non-Kubernetes orchestration options. Going full Kubernetes is rarely what organizations need. Getting CI/CD -> containers on servers and being able to configure the resources containers need, having the placement handled, etc. gets most of these folks 90% of what they want with much much less complexity.
- AWS ECS: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
- ECS Anywhere: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
Being able to mount secrets and configmaps into the container file system (without having to modify the container image to provide an entrypoint) definitely seemed to be one major advantage of kubernetes over ECS a few years back.
See a CDK example:
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_...
You can setup something to mount configuration files stored in say S3 -> into an EFS volume that you attach to all ECS tasks, for example.
The problem with ECS is that everything, including service discovery, is an integration with another AWS service. Making it even more difficult to ever migrate, but it does support much of what folks use Kubernetes for.
HashiCorp Nomad is a perfectly easy and flexible orchestrator (not only containers, also VMs, random binaries, Firecracker, WASM, etc.) that is production grade. It has a number of advantages over Kubernetes (being drastically easier to deploy, use and maintain; having integrated native templating with logic; being much nimbler but also being able to scale much more, etc. etc.) and it's a very good choice in many scenarios. Of course there's also disadvantages, most notably ecosystem size, but often that doesn't matter. Kubernetes is great and extremely powerful, but also very complex and straight up overkill for many organisations.
There is an unsupported community edition
* It can't work down batch jobs in priority order [1] -- a feature even the ultra-old university cluster schedulers had
* Sometimes job starts can be in arbitrary order, in the API and the UI, making it difficult to find the most recent ones [2]
* Nomad seems to store pretty much everything in its Raft state, making it extremely slow to startup and operate (including GUI) if some rogue process created 100k allocations. Cleaning that up seems to require nuking the Raft state, thus also losing all info about all jobs.
I don't know how well Kubernetes does in these areas.
[1]: https://github.com/hashicorp/nomad/issues/12792 [2]: https://github.com/hashicorp/nomad/issues/17742
I guess, my take away at this second is to really be skeptical of hype that I don't have years of operational knowledge and experience with. Which, to be fair, regarding my current tech du jour (well, 9+ years), I have a list of complaints that could span a short novela.
Nomad's mental model is hard to grasp. Or let's say is not as elegant. It's is nowhere near k8s. And k8s at least from a usage standpoint can be explained in under ten minutes. Pod, Deployment, Service, Ingress and you're pretty much done.
It's a wonderful, observable, manageable orchestration layer that's easily and safely customizable if you're wise enough to treat it as such.
It’s just a bug in something most people don’t use.
* Instance refresh to deploy new images.
* Every so often update your AMI to patch.
* Create an RDS database and hook the instances up to that if you need such things.
It's really hard to get more boring but works than this. It can be set up in a few hundred lines of terraform. I promise the code for your entire AWS account will be less than just the k8s YAML.
We do this but create custom images that have all their requirements installed via ansible.
Unfortunately there's now a push to move to GKE instead of GCE, for no real reason.
If your workload is extremely variable, something like k8s with auto-scaling makes sense. But a lot of applications (maybe the majority?) have very predictable loads and you don't always need new services.
When you're at the size of large corps with hundreds/thousands of services, then k8s is amazing. If you're not even close to that number, it's very likely premature optimization.
The only negatives about Nomad is the Hashicorp license drama that has happened recently and persistent storage can be a pain in the ass.
They pulled the rug out on a lot of people and people tend to hold grudges. Community edition is peanuts compared to using an open source tool you _could have_ fixed your own bugs with by creating PRs or adding features with PRs. I used to be a huge hashicorp fanboy... _used to_.
Nomad is simple on the surface and could have been a great tool - but it is basically unusable on its own without tighter integration with Hashi’s own tools (eg Vault). Configuring and maintaining all those things is nontrivial and ends up being more annoying (for a lesser end result) than just using a fully managed kubernetes cluster like GKE
However I would ask the question (same question I have been asking since the first appearance of k8s) what do you need so much that it justifies this complexity?
Implementing a simple autoscaling system or using the provided solutions like AWS ASG was and is results in more reliable infra.
If you want to use cutting edge there Lamda function that most cloud vendors offer. That is also less complex option.