Comment by peterangular | Hacker News Reader

Comment by peterangular | Hacker News Reader

Parent Full thread

I'm really considering doing this for a new spin-up myself... The stability of Debian tooling for large-scale, hyper scaleable solutions is outstanding. I just feel like the world balks at me every time I do something considered slightly "old fashioned" completely ignoring the finished product to shame me for not using buzz-word tooling.

Containerization is fantastic don't get me wrong, but I've had more success with old-school approaches to package management, deployment, optimization, debugging, etc. running thin Debian servers. Just... prod ops is easier and more stable at the end of the day. I really don't see the need to containerize everything outside of cross-platform development tooling. I also really prefer having a semblance of an OS/bash terminal when it comes to ops!

Also: this is purely anecdotal. And, to get ahead of the folks yelling "you just don't understand Docker and K8s" - yes I do. I still think they're great, I just am not fully sold on them for every use-case.

Debian plus an automation system like Puppet, Chef, etc works extremely well.

It's just not sexy enough for people to write hundreds of posts about how to set up your own package repo, understand unattended-upgrades, and do monitoring.

> Debian plus an automation system like Puppet, Chef, etc works extremely well.

Eh, it does until it doesn't. Sooner or later you run into pitfalls around the leaky abstraction of pretending your state is truly idempotent and path-independent. E.g. spinning up a new instance works fine, but the existing instances that need to uninstall a previous version to upgrade to the newer one end up breaking. Or vice-versa, existing servers work fine but then when you need to launch a new one your realize the config no longer works on a clean install and you hadn't noticed it for weeks.

Container-based systems certainly have their own problems, but it is really nice having a model where you don't allow long-lived implicit state and cruft to accumulate on your application servers in the same way.

If I don't understand what I'm doing, it doesn't matter what tools I'm using to fail to accomplish it.

Ah, the myth of the perfect programmer.

Everyone makes mistakes, and state resets such as VMs or containers are one of the easiest ways to revert these mistakes.

Everyone makes mistakes, and having real testing infrastructure is the best way to catch it.

Many mistakes are made against databases; maybe you can roll back, maybe you can't -- either way, it's cheaper in testing than production.

There are also state inconsistencies on servers. Configuration files are not updated correctly (maybe even with a silent failure), binaries are not updated correctly, temporary files left over interfere with the update, cached files do the same, etc.

There are many reasons why resetting the environment on deployment is a solid and cost effective solution to many issues.

Except when your Puppet/chef whatever aren’t set up carefully to be self-contained to be able to bring up the system to the correct state regardless of whatever interim state it might be in. It’s not impossible but it’s also not an out-of-the-box thing, it’s complicated to pull off, and usually requires constant expertise rather than being a playbook you hand off to whomever (even with expertise you can have bugs, you just aren’t in as big of a hole because you’ve ratholed on the assumption that you always have version 0 or version N-1 to get to state N).

Doing this is hard and there’s nothing in the ecosystem I’m aware of to guarantee that. Nix maybe? That’s part of the problem. The other is that docker has a conveniently large set of configs and things “out of the box” so there’s familiarity and documentation.

It’s not impossible to accomplish but building that community of doing things a sustainable way that actually addresses the pain points (rather than just dismissing it with “you’re just using the old tools incorrectly”) is what you need. If you’re going after docker your solution will have to support devs and devops. If you’re going for a niche community of enthusiasts/experts (probably more defensible and easier to grow), then focus just on a single niche use-case that general solutions could never outmatch (but don’t ignore growing it carefully if your niche solution is meaningfully better - listen to your users that you trust).

Reply on news.ycombinator.com