The container orchestrator landscape
lwn.net
lwn.net
Thanks
One small addition: Nomad recently implemented basic service discovery functionality into Nomad itself, so for simple use-cases you might get away with just a Nomad cluster without Consul running alongside: https://www.hashicorp.com/blog/nomad-service-discovery
There are many reasons to love Nomad, like its straight-forward architecture, how it plays well with other binaries to augment its functionality (Consul & Vault), super easy service mesh with Consul Connect + Traefik, and the ease of defining complex applications/architectures with the Job Spec [0].
I've used k8s a few times for work, but I just love Nomad.
The big thing for me is needing mTLS support for some of my services.
You can also go DIY with something like Traefik and just configure everything to require TLS with client TLS pass-through for enforced mutual TLS, that puts you into 'require a set of central load balancers' but you still get all the benefits.
HashiCorp has always tried to loosely couple our products so that while they work Better Together, they also work independently and allow users to choose the bits and pieces that fit their needs best.
My general suggestions for this question are that Terraform is what you should be creating your Nomad clusters with. In the cloud you can use it to define your autoscaling groups, load balancers, and use Packer to create golden images for Nomad servers and clients.
While Terraform is also useful for deploying "base" system jobs like log shippers or metrics aggregators, I generally don't recommend deploying all of your workloads with Terraform. You generally want more fine grained control over things like promoting-canaries-during-deployments than Terraform's architecture or interface could possibly support.
So now your left with how to run all of your workloads. Lots of folks hit the Nomad API or use the Nomad CLI straight from the CI runners, but Waypoint and Pack offer some useful tools.
Waypoint has more application developer oriented tooling than Pack. It's a developer experience oriented tool where Pack is more like Helm and oriented toward offering a package management solution for clusters. Since Waypoint manages building your project, it's a much more opinionated tool than Pack. You likely want to use Waypoint throughout your application development process.
Pack is much more useful if you just want to "package up" a bunch of existing code, whether your own or 3rd party, and run it in your cluster. It's intended to eventually have an extensive public registry/catalog of popular software ready to run in your cluster. This isn't an area Waypoint is concerned with at all.
Hope that helps!
Personally our team found a match using docker swarm. The reality is that we don't have a DevOps role and we needed something that anyone could deploy on
Although the general sentiment is negative towards docker swarm, its simplicity is hard to beat
This guide was useful for setting up our stack https://dockerswarm.rocks/
Are you used to reading drunken writing? :)
I read this earlier too and was very impressed at how everything was explained in plain english. I wish I could send this article to myself 2-3 years ago, it would have saved me literally days of trying to figure out what parts of k8s do what and why.
Agreed. LWN is really good at this, partly because their editors have really good "tech BS" detectors (learned this when I was writing for them). It means you really have to understand what you're saying and can't just parrot buzzwords or lingo you've read on a tool's marketing website.
Clients (aka nodes-actually-running-your-work) can be run globally distributed without issue, and we've even added some more features to help tune their behavior:
https://www.hashicorp.com/blog/managing-applications-at-the-...
Disclaimer: I am the HashiCorp Nomad Engineering Team Lead
This may be my favorite article on the subject, and I've read a lot! Others have suggested some minor corrections but overall you've done a fantastic job accurately and fairly portraying Nomad as far as I'm concerned.
I really appreciate the final paragraph as I constantly field questions about whether or not a k8s-only world is a foregone conclusion. I think there's room for more than one option, and I'm glad you noticed HashiCorp thinks so too!
I've been trying to get a MariaDB Galera cluster running on Docker Swarm for the past couple weeks and have been somewhat stymied a few things. Something like Longhorn would help a lot.
That being said they’re often glaring. So please report an issue to Nomad and the upstream of you hit an issue!
Nomad has 2 other storage features:
1. Host volumes where nodes can advertise a volume by name and jobs can request being placed on nodes with that volume available. I think a lot of folks run databases this way so they can statically define their database nodes and not worry about the numerous runtime points of failure CSI introduces.
2. Ephemeral disk stickiness and migration: during deployments an instance of a job’s intrinsic local storage can be set to either get reused on the same node (sticky) or migrated to wherever the new instance is placed.
Ephemeral disks is the oldest storage solution and is only a best effort (unlike CSI which may be able to reliably reattach and existing volume to a new node in the case of node failure). However I’ve heard of a lot of people happily using it for already distributed databases like Elasticsearch or Cassandra where the database can tolerate nomad’s best effort failing.
(Sorry for lack of links as I’m on mobile. Googling the key phrases you care about should get you to our docs quickly and easily)
Nomad also supports CNI. It's not uncommon for folks to run their network's control plane as a Nomad job and use a CNI plugin to integrate their other Nomad jobs with it. This sort of approach allows for running multiple logical networks within a single Nomad cluster (eg perhaps segments of your cluster use a service mesh while data intensive or legacy applications use host networking).
Wow. I really enjoyed this. Your writing was clear, simple and explained the landscape while navigating the marketing terms extremely well.
I wish to read other articles that you have written.
It would be great if you took some time to write about the question of climate change and energy waste.
In this regard it would be very interesting to better understand the impact tools like K8S have on global warming. Just from a pure practical POV I see many clusters burning a lot of CPU cycles with K8S processes that do nothing else than providing some kind of infrastructure for the real computing task.
I have seen K8S burning 50% and more CPU time - this with not one single payload operation, just doing its cluster thing!
This is a horrible waste of energy and we can not go ahead establishing such a bloated monstrosity as default data center OS of the future.
Not all data centers are 100% solar - I guess people in the field are LOL at this point knowing the realities. So I personally come to the conclusion that a thing like K8S is the opposite of what is needed in a world that is heading to a climate disaster. So many hours spent into optimizing the Linux kernel (successfully!) are nullified by a tool that wastes energy recklessly.
We need more than one article to help programmers of these tools understand why they should do their jobs with more love for future generations.
Also we need lots of articles about how to write efficient code that does not waste CPU cycles.
A very interesting series of articles would be "How to take down your K8S cluster" describing the many ways a cluster can be [mis]configured to easily be brought down by one single error / pod / application.
...All things Systemd does. Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone.
All the extra shit like CNI and Operators are there because K8s' own abstraction is too complicated so they had to add more abstractions to deal with it. Managed Distributions of K8s go away if every Linux distro can do the same things out of the box. "Managing fleets" just becomes running more Linux VMs or not. You wouldn't need "clusters", you'd just run a bunch of boxes and run replicas of whatever service you wanted, and all resources would be managed using normal Linux system permissions and Systemd configs. No volume mounts or persistent storage setup, it's just regular filesystem mounts and disks. Containers would be optional, eliminating a huge class of complexity (static apps make a lot of containerization moot).
And most importantly to admins - no more bullshit 15 month upgrade death march. Imagine just sticking to a stable Debian for 3 years but still having all the power of a distributed orchestrator. Imagine not having 50 components with APIs listening on the internet to be hacked. Imagine just using IPTables rather than custom plugins that change based on the Cloud provider just to filter network traffic, and not letting everything on the network access everything else by default.
K8s is just too expensive, complicated, and time consuming to survive. As soon as Systemd can network as a cluster, the writing's on the wall. The only reason it hasn't happened yet is the Systemd developers are basically desktop-oriented Linux fogies rather than Cloud-focused modern devs.
I also predict RedHat will block patches to make this happen, since their money comes from K8s. But someone just needs to make a Systemd fork using Systemd's own API and we can finally fix all the crap Systemd devs like to pretend is somebody else's bug.
So "all" we need to do is rewrite systemd as a distributed system, and make it "cloud-focused" and "modern"?
No thanks. Right tool for the right job.
It's a loop that schedules jobs. When you create a job it starts them, when one crashes it schedules another, when you delete one it stops them. Systemd already does this for a single node's service units, so all that's lacking is doing this across nodes.
The same way K8s does... it doesn't. K8s management nodes just stop scheduling anything until the worker nodes are back. When they're back the job controller just does exactly what it was doing before, monitoring the jobs and correcting any problems.
> How does systemd mount storage
Systemd has a Mount unit configuration that manages system mounts.
> or direct network connections to the right nodes
The same way K8s does... use some other load balancer. Most people use cloud load balancers pointed at an Ingress (Nginx & IPTables) or NodePort. The latter would simply be an Nginx service (Ingress) listening on port 80/443 and load-balancing to all the other nodes that the service was on, which you can get by having the services send Nginx an update when they start via dynamic config.
> How do you migrate physical hosts without downtime?
The same way K8s does.... stop the services on the node, remove the node, add a new node, start the services. The load balancer sends connections to the other node until services are responding on the new node.
This is not entirely accurate. K8s makes it possible to structure your cluster in such a way that it can tolerate certain kinds of network partitions. In particular, as long as:
* there is a majority of etcd nodes that can talk to each other
* there is at least one instance of each of the other important daemons (e.g. the scheduler) that can talk to the etcd quorum
then the control plane can keep running. So the cluster administrator can control the level of fault-tolerance by deciding how many instances of those services to run, and where. For instance, if you put 3 etcd's and 2 schedulers in different racks, then the cluster can continue scheduling new pods even if an entire rack goes down.
If you assign the responsibility for your cluster to a single "parent" node, you're inherently introducing a point of failure at that node. To avoid that point of failure, you have to offload the state to a replicated data store -- which is exactly what K8s does, and which leads to many of the other design decisions that people call "complicated".
The problem is a distributed state and consensus. It's not as simple as "just have a leader send nodes commands". There are well known algorithms and systems for dealing with distributed consensus. It might annoy you to find out kubernetes uses one of them (etcd)! Use kubernetes!
K8s is a very "opinionated" system. People think this is the only architecture that works, but of course it isn't true.
Then again, 'consistency' is a vague and ill-defined term in distributed computing theory and maybe you were only talking about a consistent distributed decentralized network with consensus.
Yes if you have the trivial case of a single node state store then there is no problem. You scale it by chucking more CPU and resources at it until it can't go any faster.
But in the real world that will only take you so far. How does your state store scale out to multiple machines? How does it handle potentially thousands or in extreme cases hundreds of thousands of writers a second? How do you keep multiple nodes in the state store consistent?
There's your distributed consensus problem.
Agreed, except I'd add 'stateless' to the microservices part. That's all k3s is good for and trying to make k8s do more is awful.
> Some bored admin just needs to make a patch to make systemd processes talk together, and half the point of Kubernetes will be gone.
Far more is needed. CRIU needs to work; live migrating containers between hosts. Network overlay, with transparent encryption, name resolution and access control are necessary. Bulletproof network storage management is crucial. Containers need to easily float among hosts, and the networking and storage they depend on must follow seamlessly.
Most of the pieces exist, but we're a long way off. I believe the future is an operating system with integrated clustering solely focused on hosting containers and the networking and storage they need.
All you need to run microservices is 1) a statically-compiled stateless app, 2) a systemd unit file to run the app and restart it when it stops, 3) load any credentials into systemd's credential manager, 4) iptables for network ACLs, 5) linux user/group permissions, 6) an Nginx load balancer to route traffic to different services, 7) journald to keep logs, 8) systemd to set per-service CPU/memory limits, 9) a dynamic DNS client to register a service name in dnsmasq when a service starts/stops. Systemd already handles intra-service network communication by handing an open filehandle to a service; alternately each service could grab its own random port and register it with Nginx along with its service name and the HTTP routes and hosts it wants to listen on.
If you shackle yourself to the design of K8s, or some implementation of K8s (K8s doesn't have all the features you described out of the box) and say "well that other system isn't K8s!", then yeah, you will find systemd wanting.
This doesn't quite roll into the 'only systemd,' but it allows you to build exactly what you're looking for.
I don’t think your solution is simpler but you are right about one thing: one does not need Kubernetes, alternatives exist.
Wait, k8s can use CRIU to live migrate containers?
For example, two operators I'm currently implementing for a client, ArgoCD and RabbitMQ Topology operator. The first allow GitOps, meaning that we update config file and the software running in the clusters immediately gets updated or configuration is being set. The second let me declare exchanges and queues like any other Kubernetes object. Now I'm sure it is possible to do the same in SystemD, anything can be done with enough engineering. But is it really the right tool for the job? in any case, the function of Operators is not to simplify and complex abstractions, this is simply incorrect.
I love systemd but it's not designed for managing a cluster of machines.
No. The main purpose of K8s is to make portable « running multiple copies across servers, routing from one load-balancing port to many ephemerally-allocated ports, collecting logs, providing a consistent control API, limit system resources, networking, secrets... ».
Systemd could do all that, but, then, it would not be Systemd anymore and would be like another K8s.
CNI isn’t some silly Kubernetes abstraction to scratch an itch.
CNI is a specification and a set of libraries for container networking.
Calico is built on top of CNI, Nomad requires CNI for multi-machine networking, Mesos uses CNI for network isolation, Firecracker uses CNI extensively. Docker does not use CNI directly but their libnetwork code depends on exactly the same principles as CNI. Life everywhere is simpler thanks to CNI.
Do a thought experiment. If you think it’s that easy, why don’t you try marrying systemd with Docker overlay networks and see how far you get?
K8S setup complexity is already highlighted but I'd add upgrade and maintenance pain to the mix as well.
I've only recently started with Nomad and while it's so simple, I question my self at every step because coming from K8S I've gotten used to complexity and over configuration. Nomad allows you to bring your own loadbalancer, and i'm experimenting with Traefik & Nginx now.
https://github.com/kubernetes/ingress-nginx
https://doc.traefik.io/traefik/providers/kubernetes-ingress/
Of course you have to route traffic to your cluster, but you are implying some cloud based load balancer lockin which just isn't true
Third option would be to expose pods directly in the network. EKS does this by default, but from my experience it's quite rare for people to leverage it.
See https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
What worked for us is to use k8s strictly for deployment/container orchestration, we have our own Envoy-based network layer on top of k8s and do not use Services or Ingress or Ingress Controllers
> Nomad allows you to bring your own loadbalancer
I think this is a widely held myth that you can't easily self-host k8s due to lack of access to a cloud load balancer. You can use your own load balancers with Kubernetes, it isn't that hard. I have production k8s clusters running on on-prem hardware right now using haproxy.
I use kubeadm when setting up my cluster and point the control plane endpoint to the haproxy load balancer.
Then when it comes to the ingress I use the nginx ingress and have it setup exactly as described in the "Using a self-provisioned edge" section of the documentation here: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/
> "seemingly every tech company of a certain size has its own distribution and/or hosted offering to cater to enterprises"
Started with 1 node acting as manager and worker at the same time. Then added more 2 nodes, basically 3 nodes all acting as manager and workers at the same.
Later kept 3 managers and 3 workers and now around 10 workers with the 3 managers.
The cluster is full of apps, celery workers, static apps or SSR.
To manage it I didn't use any UI. The CLI is really simple to use. Later to give less experienced people access to it i instant Portrainer.
It has never stopped working, either upgrading or messing around on production environments.
It just works.
The only and only complain I have id the way it mess around with IPtable rules I have. The docker swarm networking overrides them. There are several approaches, while they work, they always feel like a hack to be honest. But yeah, they work.
It's such a wonderful simple to use software that gets thing.
Also liked that encryption of the overlay networks was included.
The only thing we missed/wanted was auto-scaling built-in, but we rarely had a need to scale the number of containers.
We're thinking of deploying Docker Swarm where I work onto a VMWare cluster and basically the options boil down to sharing a storage volume between VMs (which vSphere makes difficult) and replicating storage across nodes with something like gluster (which drastically increases storage used and has performance implications).
IMHO that's an anti-pattern. Why would you run two orchestrators on top of one another? Containers also cause scheduling contention on the vSphere side, on top of the waste of CPU from the virtualisation layer. Unless you're in some very specific circumstances, run your container orchestrator on bare metal when on prem.
We did first locking to specific node, but later decided to use s3fs for those kind of needs. Never needed the shared storage critically to setup anything heavy-lifting reliable.
Although worth noting, from the beginning I had psql running within docker swarm as well, but it was locked to 1 node only, we had a good backup solution to spin up (tried it too) on another node quickly as well.
With mode nodes in the cluster, I made the PSQL into its own machine completely separated from docker environment.
Does anyone have recommendation(s) regarding books, video courses, or live in-person trainings?
Hussein Nasser too, but he has less content directly about Docker/k8s and more about networking. It's excellent though.
[0] https://www.gurulabs.com/linux-training/courses/
[1] https://www.gurulabs.com/linux-training/courses/GL340/
[2] https://www.gurulabs.com/linux-training/courses/GL360/
1. Read the book "Kubernetes up and Running"
2. After that you can start reading the https://kubernetes.io documentation
3. Alongside the documentation reading, practise what you have learnt using minikube.
4. Only after, start with Helm (which is pretty bad, but if your company is using it...)
I was just curious about your perspective-- When do you mean by Helm is pretty bad? Could you elaborate? Also, how does it compare with alternatives?
As someone who rather liked Docker Swarm (and still likes it, running my homelab and private cloud stuff on it), it is a bit sad to see it winding down like it, even though there were attempts to capitalize on the nice set of simple functionality that it brought to the table like CapRover: https://caprover.com/
Even though there is still some nice software to manage installs of it, like Portainer: https://www.portainer.io/ (which also works for Kubernetes, like a smaller version of Rancher)
Even though the resource usage is far lower than that of almost any Kubernetes distro that I've used (microk8s, K3s and K0s included), the Compose format being pretty much amazing for most smaller deployments and Compose still being one of the better ways to run things locally in addition to Swarm for remote deployments (Skaffold or other K8s local cluster solutions just feel complex in comparison).
And yet, that's probably not where the future lies. Kubernetes won. Well, Nomad is also pretty good, admittedly.
Though if you absolutely do need Kubernetes, personally I'd suggest that you look in the direction of Rancher for a simple UI to manage it, or at least drill down into the cluster state, without needing too much digging through a CLI: https://rancher.com/
Lots of folks actually like k9s as well, if you do like the TUI approach a bit more: https://k9scli.io/
But for the actual clusters, assuming that you ever want to self-host one, ideally a turnkey solution, RKE is good, K0s is also promising, but personally I'd go with K3s: https://k3s.io/ which has been really stable on DEB distros and mostly works okay on RPM ones (if you cannot afford OpenShift or to wait for MicroShift).
My only pet peeve being that the Traefik ingress is a little bit under-documented (e.g. how to configure common use cases, like a SSL certificate, one with an intermediate certificate, maybe a wildcard, or perhaps just use Let's Encrypt, how to set defaults vs defining them per domain).
For the folks with thicker wallets, though, I'd suggest to just give in and pay someone to run a cluster for you: that way you'll get something vaguely portable, will make lots of the aspects in regards to running it someone else's problem and will be able to leverage the actual benefits of working with the container orchestrator.
Oh, also Helm is actually pretty nice, since the templates it generates for you by default are actually reasonable and can serve as a nice starting point for your own application deployment deescriptions, versus trying to write your own Kubernetes manifest from 0 which feels like an uphill battle, at least when compared with the simplicity of Compose.
In regards to Nomad, however, I feel like a UI that would let you click around and get bunches of HCL to describe the deployment that you want would be pretty cool. At least when starting out, something like that can really flatten the learning curve. That's also kind of why I like Portainer or Rancher so much, or even something like: https://k8syaml.com/
> To extend its reach across multiple hosts, Docker introduced Swarm mode in 2016. This is actually the second product from Docker to bear the name "Swarm" — a product from 2014 implemented a completely different approach to running containers across multiple hosts, but it is no longer maintained. It was replaced by SwarmKit, which provides the underpinnings of the current version of Docker Swarm.
On an unrelated note, this, at least to me, feels like pretty bad naming and management of the whole initiative, though. Of course, if the features are there, it shouldn't be enough to scare anyone away from the project, but at the same time it could have been a bit simpler.