Ask HN: Has anyone moved from Kubernetes to Nomad? | Hacker News Reader

Ask HN: Has anyone moved from Kubernetes to Nomad?

If so, what was your experience?

I've been operating a Kubernetes cluster for a while now, and I'm quite happy with it. It does what I want. However, the complexity cost is quite high. That's especially true when developers who without much k8s experience need to interact with a local or staging cluster somehow to debug applications. And of course, the YAML and package management is hell despite (and partly because of) the many templating tools out there.

I'm wondering if Nomad could remove some of that complexity or whether it's still early stage and a production setup is just as complex. I'm also wondering how easy it is to deploy common applications (Postgres, Traefik, Prometheus, etc) on Nomad - easy Helm installations are a big selling point of k8s.

View on HN

we use nomad in production, no issues. Way less complex than k8s. We deploy Prometheus out of your list. We use NGINX, not Traefik. PG is deployed but not under Nomad, mostly just from laziness.

We run a local instance of send.firefox.com, redis, metabase, docker registry, a speedtest tool, a bunch of batch/periodic jobs, some internal custom services, all of our web hosting, etc directly out of nomad.

Our CI/CD runs make which builds the code, generates .nomad files and runs them, so it's super easy to re-deploy, rollback, etc.

k8s management on your own hardware is a full-time job. There are 2 of us that do all dev and ops for our product and there is 0% chance we could manage a k8s cluster w/ only 2 of us, but managing a nomad, consul and vault cluster is easy for us.

There are nomad templating tools, but we started using nomad back when it was brand new, so we just used sed for our meagre templating needs. Works a treat, and never saw any value to using some specialized tool(for our needs).

Totally recommend checking out nomad.

Thanks for the insight, that sounds very promising! Right now I'm operating a k8s cluster on cloud VMs, so not managed, but not custom hardware either, myself and like you said it's pretty much a full-time job despite the relatively simple cluster workloads. It's great to hear from someone who successfully uses Nomad in production.

What is the day to day operation needs like ?

What about managed k8s solutions such as EKS? Would they also be a full time job? I understand that is probably more expensive but I'm curious if it would still be pain as having your own cluster.

The main benefit you get from managed Kubernetes solutions is network/security, update- and scaling-related. These things are important, but they don't reduce the daily operating complexity around deployments (applications) on the cluster. In other words, they take the pain out of running and setting up Kubernetes itself, but they don't simplify using Kubernetes. I highly recommend using a managed solution unless you absolutely know what you are doing in terms of networking, security, and disaster recovery. They are not too expensive either, you're still mostly paying for the VM nodes.

The only reason we're not using a managed solution is because we have some specific edge networking and multi-cluster requirements that are hard to do with a managed solution.

Agreed.

And just to say, managing k8s on own hardware(or in cloud VM's) is a full-time position, which means you probably need more than 1 actual employee in that role, especially if you are a 24/7 shop, where you need 24/7 uptime and suddenly k8s is costing you a few full-time people just to keep k8s alive and healthy. That's a HUGE cost.

I manage nomad on the side, I spend < 10% of my time babysitting nomad, consul and vault. Basically I just upgrade it occasionally. Otherwise I rarely think about it.

I have been on Kubernetes for over a year and I'm just a team of one. Besides the initial learning curve it's never been a "full time job" like many say it is. I have good automation and I don't need to do much maintenance besides upgrades etc. I deploy and manage k8s with Rancher, and that makes life a lot easier.

Reply on news.ycombinator.com