Comment by orthecreedence | Hacker News Reader

Comment by orthecreedence | Hacker News Reader

Parent Full thread

> if you can get away with monoliths, you probably don't need Kubernetes

Right, especially if you're doing microservices the "right way" which means, each service has not only its own app setup but its own data store. I am responsible for (among other things) maintaining and scaling my company's infrastructure, and while we do have some vague concept of microservices (app, api (monolith), queuing system, various ML apis, headless browser, etc) it is always always ALWAYS "what's the simplest way to do this?" Microservices are a ton of moving parts, and adding in their own data stores is another can of worms.

We have a need for a scheduler. We are using Nomad internally. I chose this over Kubernetes. Why? I know that Kubernetes has more muscle behind it, can do more "things" we might need in the future, etc. I chose it because of its operational simplicity. It strikes a perfect balance of "scale these three things up" without needing a dedicated ops team to run it.

So far it has performed great, and I really think it comes down to limiting the number of moving pieces. Don't get me wrong, microservices are great if you have an ops team and a 50+ dev team all working on different things. But I often see microservices being pushed without warning people of the inherent costs involved.

I'm sure kubernetes is great, and I'm continuously re-evaluating where we are and what we need, but for now simpler is always better. If we do go with Kube, it will likely be a managed version so I don't have to touch the internals...not because I don't want to learn them (I definitely would want to know all the grimy details) but because I just don't have the fucking time to deal with it.

I think programming as a profession provides three main services:

1. Implementing user/client features (meeting demand)

2. Designing and maintaining an architecture flexible enough to continue adding features at a reasonable pace

3. Reducing complexity, which primarily involves picking the right primitives and reducing the number of variables as much as possible

2 and 3 are really just in service of 1, but the fact is that if you don't invest in flexibility/leverage/optionality and proactively keep yourself from getting swamped by an overwhelming number of "moving parts," your productivity will creep to a halt and formerly cost-effective features will no longer be economical, destroying your capability to provide customer value.

How do you use Nomad? Do you run Docker containers with it? I don't need the scaling, I need the part where new versions of code are autodeployed automatically, services are connected together, and basically things update without having to run ops checklists every time.

Kubernetes feels too heavyweight and deploying machines feels too snowflakey...

We use nomad a few different ways. One is system jobs which run DNS and Fabio (the networking fabric layer) on the host machines. Then, yeah, for all the apps and services we use Docker containers which I've found runs pretty well...with one caveat: make sure if you're spinning jobs up/down a lot you get a fast disk. For instance, we were running on 64gb ebs gpio volumes and Docker was starting to grind to a halt on them (we spin workers for our queue up/down quite a lot). We started using instances with ephemeral SSDs instead which has a bit more operational complexity on init, but overall works really nicely.

We also use Nomad for deployments as well. I wrote some deploy scripts by hand that create the Docker containers and load the templated Nomad jobs ie

    image = "container-hub.myapp.com/api:{{ BUILD_TAG }}"

becomes

    image = "container-hub.myapp.com/api:master.20190613-153222.f996f1bd"

I'd say as far as "take this container and run N instances of it and load balance requests at this hostname to them" Nomad has been pretty great. There is definitely some work involved getting the comm fabric all set up exactly how you want (Fabio does make this easier but it's still work). Consul now has Connect (https://www.consul.io/docs/connect/index.html) which I haven't looked at yet which might alleviate a lot of this. I think some of our complexity also came fromt he fact that we do have TCP services we needed to load balance and most fo the service fabric stuff forces HTTP onto everything.

Overall my experience with Nomad has been great. It's capable and really not too difficult to operate for one person who also has tons of other stuff going as well =].

This is super useful, thank you! Do you think all of this is worth doing, now that you can basically get managed Kubernetes for free from providers as long as you use their machines?

It feels like this is much easier than rolling your own Kube, but not easier than using the managed version...

Not sure, honestly. I've never used Kube, just taken a preliminary look at the docs and been scared away by how much abtraction there is. While providers may manage it for us, I'm not sure to what extent they manage it. We're on AWS and I haven't been super happy with the responses/response times of their support, so when dealing with unknowns I'd rather not rely on someone else.

That said, Nomad hasn't been without problems. It's just that the problems seem to be easier for one person to solve. I set all this up almost a year and a half ago and haven't touched it much since, so it's possible both Nomad, Kube, and and managed services have come a long way and now is a good time to re-evaluate.

Reply on news.ycombinator.com