Comment by tonyarkles | Hacker News Reader

Comment by tonyarkles | Hacker News Reader

Parent Full thread

I'm smiling at your comment, and please don't interpret this as me making fun of you; I'm laughing along with you at the absurdity.

> I’m somewhat struggling through getting a service working on Kubernetes. So much configuration to go through. I’m trying to build a product, not be a systems engineer.

I'm a systems engineer through-and-through. I've resisted k8s for a long time, but recently inherited a project that was already very k8s-native (for example, it's an Elixir-based app that auto-joins other OTP nodes based on k8s discovery) and figured it was time to dive in.

From my understanding, the whole point of this is to abstract away a lot of the pieces that were fundamental to "sysadmin" skills. I'm running this on a DigitalOcean Managed K8s cluster, so I haven't had to do any configuration on the nodes at all, but there's a ton of things that feel dramatically more complicated, even though they're nominally more "abstract" than they would be on bare metal. What's missing, though, are the systems power tools that help you debug what's happening! I have no doubt that this is an exceptionally powerful platform, but... when I start getting errors about a process not being able to make a TCP connection to a publicly-accessible IPv4 address, I can't just connect and look at the iptables configuration on the node to see if there's any kind of rule that might be blocking it.

In a lot of ways it feels like we're throwing out the baby with the bath water.

Yeah I’m setting this up on Linode’s beta k8s product.

Funny thing is that I know how to do mid-level server admin. Most of that comes from building and maintaining the servers for https://jsonip.com for years. iptables, nginx configs, etc. Building up a new Ubuntu server from scratch isn’t second nature, but I have a run book and enough familiarity that it isn’t hard.

I’ve even worked with Kubernetes a little over the last few years at a couple jobs. Dockerize my app, use some kubectl commands to get it deployed. Luckily I wasn’t responsible for maintaining the infrastructure.

Last night I had this realization. The last company I worked at is now publicly traded and has hundreds of millions of users. When I left they were just starting to experiment with k8s but we weren’t using it in production anywhere. There was an entire devops team working on it for at least a year and it still wasn’t ready for production.

So I realized that I don’t actually need Kubernetes. I’m super small scale and working on this project on my own. They scaled up to be a billion dollar company without it. But then it leaves me in the position of either hacking together my own limited auto scaling infrastructure or struggling through with Kubernetes.

I’m not really sure which direction to move tonight. I have enough personal funds for maybe 6 months to a year and this is just eating up time.

Do I struggle through a few more days trying to get this working with Kubernetes and “do it the right way” or give up and hack together a nodejs app that monitors system performance and auto scales for me as needed?

At least with the “hacky” app I’ll understand at a base level what’s going on. It’ll take the same time to implement, but when problems eventually crop up I’ll know what’s going on and how to fix it. But with Kubernetes I’ll forget most of the details after it’s working and fixing the inevitable issues will mean re-educating myself during a downtime. Neither is great, but it makes me lean towards DIY. And DIY is what systems like Kubernetes were built to purportedly replace.

Help!

There’s an expression at Amazon, adopted from our Indian colleagues: do the needful.

It means do what’s necessary to accomplish your objective and don’t worry so much about what the “right” approach is — that you should feel empowered to solve problems, not get caught up in the never ending stream of people yelling about what is “correct”.

Your goal is to release a product, and you have a tight deadline, so use the tools that let you accomplish that — what you’re familiar with, what you understand, and what enables you to work quickly.

If Kubernetes authors have opinions about how you work, they should show some customer obsession and release a product you can effectively use. That’s not your problem.

Your problem is releasing your app. Do the needful.

There’s no shame in that.

For some meme fun:

https://www.youtube.com/watch?v=N0fMHt2edZc

This explains a lot of the strange engineering I've seen from Amazon over the past several years.

Like why there's almost no coherence among the AWS services, or why the error-handling and documentation is never up to par, or why they're happy to throw away a library and replace it with another one with a completely different interface. Amazon doesn't need these things!

I will say that this behavior has generated absolutely zero loyalty in me. The only reason I use any Amazon service today is because I made that decision in the past, and it's more work to change. They look to me like the Blackberry of web services right now. It works, and they have millions of customers, but they simply don't comprehend how much better the experience could be.

@miscPerson you had a very good reply I don’t know why it was downvoted.

I've re-setup my baremetal server I rent at hetzner in December to have a modern ops infrastructure. I started with k8s because it seemed pretty straight forward to have it up using rancher. After about a week of not being able to configure a simple hello world nginx docker image I gave up and tried using docker swarm.

I couldn't be happier with the decision: in less than a month of working on it during evenings and weekends I was able to set up a reverse proxy with traefik, a self hosted gitlab instance, a docker private registry and a full CI/CD pipeline.

With docker swarm, if you can 'docker-compose up ...' an app, you can also 'docker swarm deploy' it, with almost no modifications needed. Also if your needs grow and you really need k8s, you can 'kompose up' (or something similar, since I haven't got there yet) and have it deployed to a k8s cluster. But I imagine by the time my apps will need to run on k8s, they'll make enough money I will afford a devops team to handle the migration.

Feel free to contact me (info on my profile) if you think I can help you in any way having it set up.

> At least with the “hacky” app I’ll understand at a base level what’s going on. It’ll take the same time to implement, but when problems eventually crop up I’ll know what’s going on and how to fix it.

Being able to understand and remember a system is what allows you to control and operate a system. If you build a simple system, you may need to intervene more often, but most interventions will be simple. If you build a complex system well, manual interventions will be less frequent, but they are likely to be a lot harder.

> feel dramatically more complicated, even though they're nominally more "abstract" than they would be on bare metal

One problem I have with most of these systems is that they use completely foreign concepts. A programmer needs basic old-school sysadmin skills anyway just to develop a program. But most of those skills, however, aren't useful when using Docker on a server. You need to speak a very different language.

It reminds me of developing for small embedded systems. You get an IDE on your regular computer to write the program, but then after you've compiled it, you have to load it onto the embedded system and run and debug it there. Suddenly, the tools are all different, and most of them are crap.

Reply on news.ycombinator.com