Hacker News Reader

Comment by lumost | Hacker News Reader

Most of the tooling that comes with linux isn't very friendly to learn, and while more transparent companies grant root access for developers, many still resist allowing ssh/sudo privileges on servers. The standard linux daemons that used to be the default logging (syslog), scheduling (cron), and deployment (apt/yum) aren't commonly used in modern cloud shops.

Too many developers, linux and other OS primitives are unneeded overhead on their application. One can write a little cloud formation, or kubernetes yaml and be off to the races while only interacting with linux as part of their build system.

View on HN

Most of the tooling that comes with a kitchen isn't very friendly to learn, and while more transparent companies grant chef access for assistants, many still resist allowing chef privileges on the kitchen. The standard oven that used to be the default cooking, baking, and frying aren't commonly used in modern kitchen shops.

Too many people, knifes and other tools are unneeded overhead on their application. One can cook a little cake, or pasta and be off to the races while only interacting with kitchen as part of their build system.

I think you can develop this analogy further; why stop? Since the kitchen usually has neither the time or the money to have a bunch of trained chefs cook their meals, they just order frozen cake and pasta from the same retailer that everyone else uses without particularly checking the quality. They also copy the menus and prices for what to serve from the internet to save time.

If the customer has a bad experience with a specific plate of pasta or piece of cake, they just say “Sorry!“ and replace it with a new one automatically. Most of the times the people coming to the restaurant are not particularly discerning anyway and just want something to satiate their hunger, so they are fine with this arrangement. New packaged flavors are rolled out periodically by the frozen food company and so the restaurant can even boast a seasonal menu!

syslog/cron/apt are overhead? They are ants compared to even the most stripped down k8s overhead.

Cognitive overhead perhaps. I hope the new simplified platform is actually good and not an ad hoc, informally-specified, bug-ridden, slow implementation of half of Unix, though.

If one isn't using the tools available from the OS or even the OS regularly, it's tough to know that they are there. I picked on cron/syslog/apt as there aren't too many times I see them used on server fleets these days. That's coming from 10 years working in both SRE/systems and Software Eng roles on fleets of 1k+ servers both in and out of clouds. It's really diffiult to convince new engineers that even having a persistent process is a good idea vs. using an orchestration service.

As for the syslog/cron/apt examples I'll share some more detail below.

- Syslog Isn't used as it's pretty straightforward to pull in ones favorite language specific log library, combined with ones favorite log tool of choice and be off to the races.

- Cron doesn't get used as it can only easily handle scheduling on a specific host. Whereas there are a plethora of distributed schedulers which will happily run commands on any number of hosts.

- Apt/yum don't get used as folks want to have their builds to be repeatable, and rely on software built in via their dependency management system of choice and deployed via fat statically linked binary.

> Whereas there are a plethora of distributed schedulers which will happily run commands on any number of hosts.

If you have any specific suggestions, I am on the lookout. I checked out tools like airflow and dask but did not see what they added. Trying not to go full blown HPC cluster.

HashiCorp Nomad[0] can do scheduled periodic tasks across a cluster of machines. Much lighter weight than Kubernetes, but a shotgun flyswatter when compared to cron.

[0] https://nomadproject.io

If you're on a PaaS solution, I'd take a look at the built-in scheduler e.g. scheduled tasks in Kubernetes, fargate or scheduled tasks in Mesos via Singularity or other frontend.

If you aren't using a PaaS, take a look at hashicorp's nomad tool. It's worth noting that if your team uses microservices or just likes making lots of services, a self-hosted or cloud PaaS solution will save you inordinate amounts of time when it's used by more than 20 engineers.

Reply on news.ycombinator.com