> I hate systemd with a burning passion
Why?
Why?
I was attracted to Linux because unlike Windows or macOS, I actually have choice with Linux. I can choose my filesystem, my libc, my init system, my bootloader, my system encryption, my window manager, etc.
Sure, it's not perfect: many programs assume that you have glibc, or dbus, or X11, or wayland, etc. Actually many programs assume that you run Ubuntu. And it is frustrating when you would want to use a program and can't because of such an assumption. But again on Linux, there is (or used to be?) a big open source culture. This program is assuming glibc? I can patch it to work with musl. Etc.
Systemd, to me, sounds like it says: "why would I care about interoperability? I am the best, you should use me. And if you want something else, you should make that something else interoperable with me, because I make the rules". It's not like systemd is trying to standardise stuff to allow alternative components to exist, right? It doesn't actively try to prevent alternatives, but that's pretty much it.
And normally I wouldn't care (again, Linux is about freedom of choice for me). But systemd being as successful as it is dramatically reduces my freedom. And systemd it seemingly trying to take over more and more components. And the more adoption it gets, the more power it has. You didn't want systemd to replace that part of your system? Too bad, it will. What are you gonna do anyway, systemd is too big to care about you at this point.
The main idea behind inits other than systemd is that stuff in linux should aim to be small and modular and systemd is a giant that wants to do everything.
I'd prefer to keep the older sysvinit init scripts, with all logs just going to text files rather than the systemd journal, but not so much that I make the effort to “fight” to keep things running that way on new installations. A lot of stuff still works the old way anyway (journal logs being mirrored to files in /var/log, init-like scripts in /etc/init.d calling systemd parts to mimic what the script on its own would have done previously so you can still `/etc/init.d/apache reload` or such).
Really, the thing I like about pretty much all the others is that they don't reduce my freedom. Systemd does.
- Too many parts.
- Doing too many things, sometimes poorly.
- Fights with other services, no graceful fallback & logging (i.e. aggressive).
- As a result, unnecessarily complex, brittle and fragile.
- Harder to secure (at code level) than simpler tools.
- Project is too aggressive and want to replace any and every service.
- Sometimes break in unexpected ways.
- Not faster than Parallel Sys-V.
I can remember having to deal with SysVinit and thank god for Poettering and all the other systemd maintainers for bringing the Linux ecosystem to the 21st century
I can understand the frustrations with Sys-V init scripts, but on the other hand, I never had a problem with that both as a sysadmin and a distribution developer.
To be honest, loginctl allowed us to pull some hacks, but to be fair, that hacks were to handle bad behavior from a very bad codebase. Any self-respecting development team wouldn't have written anything which needed OS/kernel level reaping to clean from system correctly.
We have been chatting recently, and someone who's more experienced than me said that they have never seen Nixie's cron to die spontaneously in the last 30 years. systemd's timers not only did that, the systemd didn't bother to restart it. Ironic, considering that the biggest feature of systemd is service supervision and unattended recovery.
I'm not that much of a sysadmin, but as a Linux user and home lab owner, I definitely prefer the integrated systemd solution. At least I can be certain that the interactions between the components are known and tested. If you instead throw together loads of KISS tools in the old UNIX style, all your tools might be individually correct but their interaction is just bad.
Learning new things is not a problem, but now I have to adapt into that thing and it might not even perform the same, and a fallback might not be possible. Opening a great can of worms you didn't want in the most unpleasant time.
One of the problems with integrated solutions is how much glue code they contain to handle these interactions. KISS tools talk over sockets, via very simple protocols. i.e. they shout each other over mountains using well defined horns and the voices they use are distinct and simple. Less glue code, simpler codebases, more robust tools.
KISS, Everything is a File, Send conservatively but receive generously are three pillars of what makes *NIXen robust systems. If that interconnectedness and meshing of things were robust, we wouldn't have made Windows a laughing stock for its crashes and security problems.
Tell me, how can a WMF or cursor file can infiltrate a whole OS? How can not you print without installing half of the desktop or literally can't have a terminal only Windows Server system? Because everything is connected and they can't be separated anymore.
These "tools thrown together in a bucket in haphazard manner" have kept systems running for literal years between reboots because they talked over these simple sockets. Heck, many good Linux servers die not because of software, but hardware failure.
And I respect Linux because this is still true despite systemd today.
Oh good _lord_ you absolutely cannot. It has been my professional experience that Systemd Project is fractally complex, most of that complexity is accidental, and approximately no one on the project is interested in investigating and fixing the intermittent-but-showstopping problems that arise from that accidental fractal complexity. [0]
Poettering and crew really built the project a fantastic escape hatch with their whole "Your usecase is legacy and we're not supporting it." attitude.
[0] The mess of tools in the Systemd Project work fine until they very suddenly do not, and then the best you can do is bodge a workaround because you're very unlikely to ever get a fix for your problem. The folks that staff the project simply don't have the attitude required to work on and maintain a ever-more-sprawling project that aims to be _the_ userspace thing atop which every Linux distribution is built. "That's hard to fix/do. I don't care that you've had a working system for 10+ years, I'm not fixing the thing that's broken for you." is an absolutely unacceptable attitude for folks involved with such a project to have. [1]
[1] If the project was a little take-it-or-leave-it side project, that attitude would be entirely acceptable... encouraged, even! But because of the position in which the project willingly and aggressively put itself into, it's not a little side project.
All of those components you mentioned have improved my linux experience on both servers, desktops and mobile clients.
So, experiences differ, apparently.
Off the top of my head, I've used both systemd and non-systemd for: resolver, timesync, networking, logs.
My main issues stem from it's prevalence in server Distro's especially for micro service images.
D-bus is one of the great offenders for me and I don't like having it on even my desktop system but systemd wants it for many things and some things will just break without it. On a small micro service image, it would be nice to be able to strip lots of it down but systemd fights you on this because of the amount of moving parts and interconnected subs-sytems it introduces.
I am a nixos user so have to use systemd on nixos and that makes it very painful, but I can see myself moving away nixos as my host machines os for many reasons but systemd and the painful way nixos uses it is a big one.
Sure I also don't want to take away from what nix and even NixOS does do well, it is a great project and has many great ideas.
Also not sure you are aware (I think you probably are but I will state it any way) Nix != NixOS, and to complicate things further Nix it's self has many parts, it's a build system, a language, a package manager, enviornment profile manager (think like what python venv's provide) and kind of what ever you can make it do.
I will say Nix as a tool is a fantastic thing to have around and now with AI the wrangling and learning can become less painful but it does allow you to do a lot of cool things. Building Deterministically Deterministically building environments, integration tests/ setting up environments for such tests, more easily managing multiple possible builds for a project (well easier then wrangling make files or what ever directly and does provide nicer abstractions (not pain free) for building code). If nothing else give nix a try for building your next project and see what you think, very nice for setting up your dev env as well.
Personally I use something called flakes which is a way of managing a project and pinning it's deps. And this is one of the points things will break down and you will understand some of the issues. flakes are supported and used by many but also many reject them and the tooling is just kind of their and breaks some bits of nix. This aspect of different competing community standards is very very common in the nix world and is one of the major pain points.
> I worry that, since there are so few who truly understands the internals.
I have not filter it in that way but, I am not sure the issue lies with the lack of knowledge of the internals but nix seems to complicate things... a lot.
You have the nix daemon and nix store. Until you have good working knowledge of how nix works (not necessarily knowing the code I have only glanced at the internals of daemon and such and the more important aspect is actually the code written in the nix language generally). Don't touch store directly including trying to copy files manually between stores and such the daemon keeps a sql database of the entries and other things about the nix store means you should not directly touch it until you understand this and it's not intuitive or made to be understood easily.
>NixOS? Finally NixOS it's self is great especially at the start and once you get it but the whole idea is declarative configs and well I know have spent lots of time fighting this and while it does have escape hatches they can be obscure and confusing and error prone. An example of this might be something like llama-swap module, you configure your modules in nix and llama-swap will module will configure a systemd unit for you once enable it which sounds great but in truth you probably want to do things outside the config and make ad-hoc changes and you can set it up so all this is possible but you end up finding your self avoiding a lot of the declarative features and having to edit your config then build it and push it every time for things that are small can be very annoying.
The thing is though you can generally just tell it to use a path to your config or give it a config or even not cofigure a systemd unit at all, but many of these things stack up and you find your self fighting it a lot of the time. The default will have things like a firewall unit that pushes some 'useful' firewall rules that block listening and such and works with declarative configs to enable particular ports which is good, but if you don't know that you might spend a bit of time wondering what the hell is going on.
I would say if you still interested look first at nix tooling and nix and if you want to dip into the operating system you will understand a lot more about why things are the way they are and if you are someone who is finding they want to make a lot of quick ad-hoc changes to the system in question nix can get very annoying. The language is not great and difficult to learn, the main issue is that it's not typed and a lot of the features and right way to do things is arcane and occult, AI will help learning all this a lot though and bare in mind it's a lazy functional language.
Alpine for me is looking more attractive to build my host system on and still use nixos for some dev vms and other vms I use and configure Alpine in a well defined way but still lets me quickly hack a bash script into something. Again you can use nix it's self on any distro and even mac and windows (I think).
Just wanted to take a jab. Feel free to downvote. =]
If the command is really instant you can do "command & sleep 30; command".
heh.
* * * * * user dothing
* * * * * user sleep 30 && dothing
If the task might sometimes take 30 seconds to run you have extra faf debouncing it to avoid race conditions, with this solution or yours, which IIRC systemd timers will deal with (by being configurable to not run the next instance if the previous has not yet completed).
But do you run systemd and complain or why do you even hate it?
its linux right, if a linux distribution decides to use it, its not systemds fault to be used and if you use it and don't take the time and effort to have it your way, you shouldn't complain either?
One of the most important feature of prior tools were transparency. Since they were single-purpose tools written with UNIX philosophy in mind, they were simpler, more robust and has less fail-modes. Systemd is pretty opaque and non-deterministic in that regard. This is not a "old man yells at cloud" stuff. You can't trust to a piece of code you can't trace and debug. Systemd makes it almost impossible with all the intertwined gears and IPC layers it has.
On the "I didn't have any problems" with it side. It might be working on your system, for your use case. It might not show its bugs on a system with a 90-day average uptime. I don't live on that world. I manage way more systems with way more different, niche, interesting hardware and software than that, and this tends to surface edge cases and where things break.
On the "do you run systemd?" part, yes I run it. On many, many machines 7/24/365.
It's easy to say "Why do you even run it, it's Linux, just distro-hop", when you have a single system. I don't. Projects, software has requirements, compatibility targets, etc. Also, considering the hardware we run, it's not that simple either.
Lastly, I run Debian personally since version 3. We use other distributions alongside Debian in our datacenter. I'm very involved and rooted in these systems, and Debian feels like home after using, contributing, and developing derivatives on top of that.
...and how systemd has been forced into systems is another story. I have been there, watched that, so, errm, it's not all voluntary, peaceful and friendly, to put it kindly.
Been there too. The thing I disliked most, that it sometimes get stuck in some strange pose, without any reasonable way of remediation. And because it is so annoying to debug, reboot become the only tool do deal with it.
The sheer complexity of systemd created nasty feedback loop: it is too complex to be debugged; people prefer to reboot it; that keeps problems keep accumulating inside.
We are no longer 50 years ago were a linux system or distro was fitting on a floppy disk.
The handful of tools like grep, cp, etc. have a complexity of a banana and everything else successful is so big, that it doesn't fit at all.
Blender, container, ffmpeg, vlc, vim, firefox and co. Huge code bases. And if someone woudl argue "yeah but firefox is doing one thing, its browsing" thats an argument you can bring for systemd too.
Systemd is just not a 'start services in the right order' tool. Its more holistic
All the examples you give fit into UNIX philosophy rather well. Firefox doesn't do text editing and ffmpeg doesn't render web pages, and vim doesn't transcode videos. They do one thing and do it well. They are as simple as they need to be not more.
Also, if GNU coreutils have a complexity of a banana, why uutils still can't replicate them as-is? Why they had ~140 CVEs in one release despite being written in Rust? If grep is simple, why we have only two performant implementations, grep and ripgrep?
Even FreeBSD guys discussed this: https://lists.freebsd.org/pipermail/freebsd-current/2010-Aug...
You're simplifying too much, I believe. Try writing a cp clone, why not?
systemd started as a start services in the right order tool, then bloated from there. Being holistic is not the problem, being holistic in the wrong way is.
Now, they decided that DBus is not nice enough and created their own IPC mechanism. What was that? Varlink? They don't stop there even. They say "Hey, why not try Varlink?". So to translate: "We want to kill DBus so, we will force Varlink on you".
Just starting a service at boottime doesn't help much and ignores a lot of stuff around.
What do i start when a user logs in? how many resources is the service allowed to use? How do i control that service and read its logs?
If i already start a service, i can also start a service based on other events like bus events (which are needed potentially anyway for services to communicate) or by time.
And all of this needs to shutdown properly when finished.
Actually, I would argue that browsers are doing too much :-). Webapps shouldn't be a thing, browsers should be for loading websites. Apps should run outside the browser in the OS.
You did not deal with the most important complains that is the easy of debugging, modifying and changing stuff. It lacks a lot of this and one can question how much of an improvement ini like files are that are then interpreted in opaque ways that to provide desired features such as containerization or jailing. I personally want to usually run containers with podman, do systemd containers give me anything or is it just more cruft, same with isolation, do I really gain much from the complex namespace abstraction rather then using unshare.
> "yeah but firefox is doing one thing, its browsing"
Browsers as they exist today where a mistake that is clear. those other tools you mentioned might be large but they don't violate do just one thing well.
I am not even that much of a unix guy (I use emacs which the unix people like to shit on of course and does not fit with the do one thing well). But you are not providing arguments against the issues we have with systemd and have not demonstrated its value over other init 1 systems and even the harder case you seem to argue for it's applicability everwhere.
For me systemd is similar to blender, firefox and co.
But the unix philosophy thing came not from a time of blender, firefox and systemd but a time of ls, cat, grep, xargs and co.
I explicitly said firefox is not like this (and browsers are bad) and I don't really see your point with blender (though don't use the program so I can defer to you knowledge), I even said I am a heavy emacs user and so don't believe in dogmatically following 'do only one thing well' principle.
All you have to demonstrate is what is the value.
> But the unix philosophy thing came not from a time of blender, firefox and systemd but a time of ls, cat, grep, xargs and co
are you trying to say ls, cat and grep have been superseded? this is absurd these tools are still useful and not replaced what is your point here what are you trying to establish?
My question would be where is systemd a better experience or easy to work with then openrc, have you ever had to disable any of the systemd defaults services such as resolving or networking and had issues with the way a distro set things up. I don't know how you can claim you have never an into an issue with it given that any piece of software you will run into issues with.
I am also arguing more for openrc which is a newer project that has notions of dependence's and such.
If I may show where I think 'do only one thing well' is violated and correctly so is emacs (at least in part), for by having emacs as what boils down to a constant text interface I can use generalized text editing, searching and deriving buffers all in one program not having to configure anything else, my editor is my console and I can use my editor tools in my console even the ones I make.
So where is it demonstrated that the coupling is a boon that out weights the cost.
unix fixed MULTICS which on 70s hardware was already too complex. The unix philosophy is applicable to problems of whatever complexity and is a mere instantiation of problem solving strategies that humans and even crows have been using for ages.
It's difficult to avoid systemd at this point and it's not because it's the best technical solution for all use cases in the same way that modern windows is a complete shit show but still everywhere.
My question would be what does systemd do that is not possible outside of openrc and launching what ever daemons you want for the tasks you want or combining it with what ever programs you want to do things like sand-boxing or containerization? Not saying openrc is a panacea but systemd is definitely not either and for many systems is just plain annoying, I can consed for people who want a normal desktop (plug and play distro experiences) experience it can provide that.
As I said, i'm using it for probably a decade by now.
The thing with the whole systemd hate thing is everyone now knows why people might hate systemd (I am not a fan). I think a lot of features around isolation and systemd containers where useful but now probably using your containerization system that best suits your needs.
For me when I get really mad at it is when I want to build small nixos Images and want to minimize the footprint of systemd and run few services (some of that is on nixos as well).
I think it's better suited for desktop systems then server systems and it's quite annoying how prevalent it is. D-bus is one thing that always annoys me and how everything in systemd wants to use it and you can run into issues disabling it (Also don't like D-bus even on a desktop system).
Overall I think and hope we will see a rise not necessarily in openrc but in simpler init 1 systems that try to do less and provide good strong primitives that allow for easily using what ever daemon configured how you want, rather then trying to provide coupled implementations for everything.
Again I know these criticism are something of a meme, but they still are true.
systemd (or rather Poettering) also isn't really a fan of dbus. That is why most new IPC APIs are using Varlink (and socket activation) instead of dbus. They've also been slowly migrating some of the internal IPC calls to Varlink instead.
Systemd probably is never getting rid of the dependency of dbus due to providing public APIs via it, but maybe it can be downgraded at some point to an optional dependency where APIs are registered if it is installed and just skipped if not.
True and he is right, but it's still a dependency on systemd, Varlink is some ways better, (giving much more control over listening and control over what receives messages) but the point still stands I don't want my init system to depend on this kind of userspace IPC mechanism or have to provide this, especially on systems I want to lock down.
In the resolver, in the ntp service, in the logging system, pretty much every part has had them and they keep coming. Largely because they introduce complexity that most do not understand. Some unfortunate decisions have not been recitified until the original developer lost interest and their heir had to make a heroic effort in figuring out the code.
Let's also not forget that the only time in history a major Linux distribution was backdoored, was only because some openssh package maintainers felt the need to link sshd to libsystemd. This was obviously not directly the fault of systemd maintainers but the culture around it absolutely contributed.
Only Rube-Goldberg complexity on the scale of systemd made the XZ backdoor possible. Systemd was made by despicable individuals who went on to work for despicable companies (as a big fuck you to open-source lovers) and then later on to work on even more despicable surveillance project (as a big fuck you to humanity).
systemd has its tentacles meddling in every single part of the Linux ecosystem.
But thankfully there's light at the end of the tunnel: thanks to VMs and containers and the need for speed, we've seen many minimal distros (I'm thinking stuff like Talos for Kubernetes) and those of course don't use systemd.
As soon as I get the time to switch away from my current hypervisor (Proxmox, so Debian, so systemd) to an hypervisor not running under Linux, like maybe bhyve on FreeBSD, I'll have all my VMs and containers and hypervisor totally systemd free (my VMs and containers are already systemd free: I did in anticipation of, at long last, giving systemd the middle finger).
But of course none of this matter much because according to HN we're already at a point in time where we can write: "Plz rewrite an init system".
The real question is: why don't people hate a rube-goldberg init system made by someone who went to work for Microsoft, while using crypto made up by his brother (certainly secure we're sure), and who then went on to create a full-on surveillance company working on locking down hardware?
Regain control of PID 1 people.