ancieque

Mirantis still offers docker swarm EE.

Pretty much the same as our goto for projects at work: Hetzner + Docker (Swarm) with some Ansible to orchestrate things

We have built some automation around cluster management over at https://github.com/neuroforgede/swarmsible.

I used to do everything in ansible, but Docker Stacks are just so much nicer to use.

In any case automation is king. I don't have to remember stuff if I can just look at some IaC Code :).

Have you thought about using a single node Docker Swarm? That gives you benefits such as healthchecks for containers so unhealthy containers are replaced, support for rolling upgrades and some other goodies like secrets and configs managed by swarm.

make that another +1

So. This happened. While playing around with OpenFaaS I noticed that not much had to be changed to get the deprecated support back on track.

This is experimental, but maybe interesting to some.

Swarm never actually went away though.

The dust is settling and people are starting to realize that Kubernetes is too complex for a lot of cases. This is where things like acorn.io try to live, but this all feels like trying to get back to the same experience Swarm had.

I learned about a lot of things by watching videos by Bret Fisher. He has a lot of good resources on running Docker Swarm

pat_on_back(ThrowawayTestr)

About the order. I mean the order in deploy

``` deploy: mode: replicated replicas: 3 update_config: order: stop-first parallelism: 1 rollback_config: order: stop-first parallelism: 1 restart_policy: condition: on-failure ```

For the volume plugins:

We are using Hetzner, and this one works great: https://github.com/costela/docker-volume-hetzner . Also, there exists one for glusterfs (https://github.com/chrisbecke/glusterfs-volume).

If you are interested in making this work with any of the constraints, I am sure that there is a way to work around all these issues.

About [0]/[1]: I guess you are right in this not working out of the box, but this could possibly be worked around with a custom entrypoint that behaves differently on which slot the task is running in.

> (Since when you restart a stack to add the new service, the old service will also get restarted. If I'm remembering what I found correctly.)

Are you sure the Docker Image digest did not change? Have you tried pinning an actual Docker Image digest?

> Then there's updating services/replicas. You cannot have Swarm kill a service/replica until after the replacement is actually up and running. Which means you'll need to create a new volume every time you need to upgrade, otherwise you'll end up with two instances of your app using the same data.

Is this true even with "oder: stop-first"?

> To complicate things, as far as I can tell, Swarm doesn't yet support CSI plugins. So you're pretty much stuck with local or nfs storage. If you're using local storage when deploying new replicas/services, you better hope the first replica/service starts up on same node it was on before...

True, but there are still some volume plugins that work around and local storage should work if you use labels to pin the replicas to nodes.

My opinion on this is that if you need at least once semantics it is best to encode this behaviour in a store you can trust, take the ID of the task and then schedule a task with that data.

Then, add a "sweeper" that checks if the task actually happened and if it did not happen, requeue the job.

To ensure proper behaviour that the same job is not being worked on twice, add some locking in there and you have a really stable way of scheduling jobs.

Could this have been done INSIDE celery? Sure, but I would say that this is not the case that it should optimize for.

Could you elaborate on the quirks regarding stateful apps?

Docker swarm has Job support nowadays, would this help?

Hetzner has been offering this for a while. Worth a shot :) https://www.hetzner.com/dedicated-rootserver/mac-mini-m1

Have you tried out the "Serverless, but still Containers" offerings they have (Google Cloud Run et al.)?

Which cloud providers are you using mostly?

docker-stack-deploy is not adding too much, and I don't see it doing much more in the future.

But I get your point and believe it is in good faith, so thank you for your thoughts :).

It does what it should for us.

SSH + "make deploy" is a really bad comparison as you don't get overlay networking for free and also don't get any HA.

What we need:

1. HA for services 2. networking between nodes 3. ingress into the cluster 4. possibly volumes

All 4 are solved to our liking. Once we really need Kubernetes, we can do the switch.

It works and is simple.

This tool is a really simple layer around docker stack deploy that caused us some headaches for automation.

We are building our internal tooling and app deployments around Docker Swarm if possible due to the low barrier for entry. Clustering concepts stay the same anyways once we need to switch.

And while we are doing so, we take older tutorials and versions for swarm and modernize them on our GitHub. For example, see our modernized swarmprom here: https://github.com/neuroforgede/swarmsible/tree/master/envir...

Story of one of the projects I am involved in:

We came from Ansible managed deployments of vanilla docker with nginx as single node ingress with another load balancer on top of that.

Worked fine, but HA for containers that are only allowed to exist once in the stack was one thing that caused us headaches.

Then, we had a workshop for Rancher RKE. Looked promising at the start, but operating it became a headache as we didn't have enough people in the project team to maintain it. Certificates expiring was an issue and the fact that you actually kinda had to baby-sit the cluster was a turn off.

We killed the switch to kubernetes and moved back to Ansible + nginX + docker.

In the meantime we were toying around with Docker Swarm for smaller scale deployments and inhouse infrastructure. We didn't find anything to not like and are currently moving into that direction.

How we do things in Swarm:

1. Monitoring using an updated Swarmprom stack (https://github.com/neuroforgede/swarmsible/tree/master/envir...)

2. Graphical Insights into the Cluster / Debugging -> Portainer

3. Ingress: Treafik together with tecnativa/docker-socket-proxy so that traefik does not have to run on the managers

4. Container Autoscaling: did not need it yet for our internal installations as well as our customer deployments on bare metal, but we would go for a solution based on prometheus metrics, similar to https://github.com/UnclePhil/ascaler

5. Hardware Autoscaling: We would build a custom script for this based on prometheus that automatically orders servers of Hetzner using their hcloud-cli

6. Volumes: Hetzner Cloud Plugin, see https://github.com/costela/docker-volume-hetzner - Looking forward to CSI support though.

7. Load Balancer + SSL: in front of the Swarm using our Cloud Provider

Reasons that we would dabble in k8s again:

1. A lot of projects are k8s only (see OpenFaaS for example)

2. Finer grained control for User permissions

3. Service Mesh to introduce service accounts without requiring to go through a custom proxy

I would recommend Bret Fisher tutorials https://www.bretfisher.com/

see https://github.com/chaifeng/ufw-docker

After reading the linked article in more detail, I might have misinterpreted things a bit, but my concerns are still related :)

Yes, 100% this. Too many frameworks claim that they are "easy" just to find out they left out things that other older frameworks have already solved.

While we are on this: Do you know of any task scheduler framework that is similar to celery, but has better guarantees around task execution than what acks_late= True gives you?

I always find myself building a system that stores the really important Tasks in Postgres so that I can recover from anything in the broker or celery crashing. What i use celery for is just scheduling these Tasks by creating a celery job with the Postgres Job ID as the parameter.

Then, to detect if something went horribly wrong, I have a sweeping job that checks if any job in Postgres has not run in celery for some reason. If that is the case, we just re-queue the job.

I don't think it will go away any time soon. If the current feature set is enough for you stick with it.

The next release will even include CSI support it seems, which will be a big longevity booster.

Great resource, but some of the things it references are out of date. To get into Docker Swarm with a solid understanding of how to do things, I recommend Bret Fisher on Udemy.

His Talks are also a great resource:

https://www.youtube.com/watch?v=V9fxU5zJKb4 https://www.youtube.com/watch?v=5zY5_iTGIsU

CSI might be happening this year though :). All the big PRs were merged recently.

Now we wait for the releases.

Not out of the box, but it should be scriptable.

What is this? An argument for actual decisions based on merit? That's not the cargo cult i signed up for!