“Support for System V scripts now deprecated; will be removed in future release” | Hacker News Reader

“Support for System V scripts now deprecated; will be removed in future release”

github.com

29 pointssweettea141 comments

I continue to be baffled by people objecting to systemd. I know some people just don't like it and that's fine, people make alternative distros without it. You're not alone.

But I just have to ask, even if it sounds condescending (I'm sorry- but I really am just honestly baffled-) has anyone who complains about systemd actually written a unit file vs an init script?

I have used init scripts, I have written init scripts. I hated it. Every program needed it's own little way of doing things and writing an init script for a daemon was a hassle of bash scripting and copy pasting.

Now you can get so much functionality out of literally just an ExecStart with a few lines of boilerplate. It's beautiful. And all your logs are in journald, which for me get scraped by a single promtail config into Loki. And systemd's structured logging means I get a lot of metadata available by default for filtering and tagging.

It's just swell. And it's free software!

Of course we’ve written both. I don’t even care if it’s better. Large part of the value of Linux for me is stability and lack of churn. I’d just as soon learn another OS if I’m going to put in effort. And I’d way prefer plain text files in /var/log I don’t need weird tools to view. Don’t even know what prom tail or Loki are and I’ve used Linux daily for over 25 years. Blech. Was all part of red hats selfish move to where they are today. Hoping they go down hard.

I mostly like systemd but journalctl/journald unequivocally sucks arse with passion.

/rant

They made binary format THAT DON'T HAVE ANY FUCKING PROPER INDEXING. It might have actual indexing, who the fuck knows, but it don't have any useful one

They could literally just drop in SQLite there and now your entire's system's logs have power of being queried via SQL which would be amazing and easy to use with countless tools that can be used with SQLite but NO, (probably) Lennart wanted to have some fun writing shitty binary format coz of his chronic NIH.

    strace systemctl status 2>&1 |grep /var/log/journal |wc -l
    71

It opens SEVENTY ONE FILES TO GET LOGS ABOUT ONE APPLICATION

    systemctl status tinc

FOR WHICH IT HAS NO LOGS IN THE FIRST PLACE

    journalctl -a -e -n10 -u tinc |cat
    -- No entries --

(it's the same for app with logs btw.)

I can literally grep a bunch of text files faster than it can look thru it's "efficient" binary logs. But hey if you waste RAM to cache every journal file on your machine in RAM instead of, dunno, actually useful stuff, those commands are fast...

It's an abomination and affront to good engineering. Which is shame coz I like many of the other parts.

/endrant

> Large part of the value of Linux for me is stability and lack of churn

Systemd is 13 years old.

As far as I'm concerned, the complains against churn run out after the project is old enough to drink. Especially if it's not just well adopted, but by far the most popular option.

In general, having open source but being afraid to change it sounds like hell to me. I personally believe in Linux & open source because it lets us keep improving & iterating & changing. Open source embodies the soft in software, and is the best vector computing society has for bringing light & clarity to a confusing but powerful man-made microverse.

Linux is an IBM product at this point. Might as well use AIX. Makes me sad, but it’s done. With what they did to kill centos and rocky I’m switching our stuff to open BSD. Pushing these changes were strategic redhat business decisions masquerading as open source contributions.

As a Debian user, it doesn't affect me. Supposedly back ported sources are also available just no longer tagged for RHEL, and if that's true my sadness is somewhat limited for my fellow distros, even if they are so heavily corporate in origin. My world & the Linux world keeps getting incredibly better every year, in my view.

This 254 release is amazing, just so much stuff. Utterly unreadably long but posted some favs: https://news.ycombinator.com/item?id=36516792

Chin up, man! Or best of luck over in openbsd land.

What and where were you drinking at age 13? (-:

I've been using it (including writing unit files etc) since archlinux adopted it, so over 10 years now.

I still don't like it.

The binary interface to logs is a good example, not only for the popular dislike for needing to run a binary to view the log, but also, central to my dislike, is the issue with the "init" system being in charge of logging at all.

systemd if far far beyond an "init" system, it is striving to be the "kernel" of user space, in charge of all system functions that are not in the kernel.

This isn't feature creep, it's by design. Is the "init" system in charge of network name resolution? What about making fundamental configuration setting to the OS? Another example of the binary required to access logs, is any change to a non-running root filesystem. If you've configured a rootfs for a machine other than the one you're doing this work on (maybe a different CPU architecture, or simply a rootfs for another machine) you're going to run into complications. Same situation when in a chroot. If the rootfs you're configuring, isn't the one currently active, those binary executables are going to havee trouble.

As soon as we have systemd-packagemanagerd, there will be 0 difference between systemd based distributions. Everyone will simply put the kernel together w/ systemd for a complete operating system. This was Red Hat's intention, now IBM's intention.

I use linux for my work, both in embedded targets and in my development workstation. For many years I've simply copied my home dir to new computers and moved on. The next time I configure a workstation from scratch, I'm going to voidlinux. runit, based on The DJB Way, is much closer to my ideal...

An OS has a lot of things to be done downstream of the kernel in userspace.

Having one interface to them is useful.

i respect the BSD hacker style, minimal init ethos.

But man is it handy to have declarative sandboxing in service files.

`loginctl` as a standard for listing, locking, killing user sessions.

`machinectl` as a container / virtual machine manager.

And --host / --machine to remote all of the above.

For the umpteenth time, systemd is not a single thing, it’s a group of related projects. The core is just big enough to fulfill the (quite hard) job of bringing up a system, doing dependency resolution and running services. Plenty other parts of the project are individual services only.

Is KDE “longing for world domination” because they have multiple projects as well?

> Large part of the value of Linux for me is stability and lack of churn.

over which part of linux's history do you feel this happened?

during the years long libc migrations? during the thread wars?

which linux-based OS have you used for a couple of decades that hasn't had large changes but has managed to continue to exist? slackware?

https://github.com/awfulcooking/journalfs

> I have used init scripts, I have written init scripts. I hated it. Every program needed it's own little way of doing things and writing an init script for a daemon was a hassle of bash scripting and copy pasting.

We had, no joking, thousands lines of init scripts that were just ripped out of distro (mostly CentOS/RHEL... Debian had some pre-made wrappers that mostly did the right thing) and fixed by us because people saying "it's so easy" to write init script ARE LYING.

And some apps, like Pacemaker (app for making other things HA), are really sensitive on scripts not being done fully correctly

Some tidbits:

* script reports app is stopped right after you run start... because it left creating PID to application, app took some time to boot (Java), and the script didn't wait for PID creation. So running "start, then run status to check whether it worked" returned that app did not work.

* script reports app is running after reboot. The app is not running. Script read off PID file off /var/run (that at the time wasn't in tmpfs like now /run is), saw PID exists and didn't try to start it at all. The pid was of a different application. That was MySQL, not some random obscure app...

* "stop" action doesn't actually return when app is stopped, it just sends signal. sysv goes ahead and shuts down system before app manages to write all of the data. Or in case of pacemaker it does weird stuff because "status returning running after sending stop" should not happen.

* other various ways "status" was broken randomly

OTOH "systemctl start httpd.service" exits with 0 even if apache doesn't start because for some reason "a misconfigured apache didn't start" is considered correct behavior. I don't know that any init has actually solved that problem.

> OTOH "systemctl start httpd.service"

I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file.

Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl. I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.

> Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl.

I mean it's PITA but you can just add strace command to start of ExecStart...

> I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.

Yeah we hit that with some units that had maintainers preemptively setting security policy so distro-blessed paths were only writable ones for a given unit.

But on plus side all could be fixed in few lines of overrides vs putting whole init script into the repo

> I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file.

Do you have systemd unit for that mysql problem ? There is option to redirect stdout/err to logs directly but systemd is too stupid to extract that logs back and show it in the status command.

I definitely saw that issue you mentioned (it almost looked like it wasn't logged anywhere if app exited too quickly). But I don't see it on recent debians at the very least.

    ● mariadb.service - MariaDB 10.5.15 database server
         Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
         Active: active (running) since Mon 2023-05-08 12:41:26 CEST; 1 months 21 days ago
           Docs: man:mariadbd(8)
                 https://mariadb.com/kb/en/library/systemd/
        Process: 2913787 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
        Process: 2913788 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
        Process: 2913790 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemct>
        Process: 2913887 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
        Process: 2913889 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
       Main PID: 2913870 (mariadbd)
         Status: "Taking your SQL requests now..."
          Tasks: 18 (limit: 2340)
         Memory: 303.5M
            CPU: 10h 32min 54.873s
         CGroup: /system.slice/mariadb.service
                 └─2913870 /usr/sbin/mariadbd
    
    May 08 12:41:26 hostname systemd[1]: Starting MariaDB 10.5.15 database server...
    May 08 12:41:26 hostname mariadbd[2913870]: 2023-05-08 12:41:26 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 2913870 ...
    May 08 12:41:26 hostname systemd[1]: Started MariaDB 10.5.15 database server.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysql' as: /usr/bin/mysql
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: You can use --force if you still want to run mysql_upgrade

> Do you have systemd unit for that mysql problem ?

Yeah: whatever ships/shipped with Debian. (Which we used at my last job, where I had this issue more than once. Don't run the same stack at my new job.)

Neither does some init scripts we had to fix so it's 0:0 on this particular issue.

That issue is actually systemd considering success if config and dependencies are okay and service is started; it doesn't wait to return so it doesn't know whether service "works okay now", because it might exit with error 20ms after start.

    Type=notify

fixes that issue but obviously needs support from app themself to signal "I'm up and running now"

Less robust way is to set

    Type=forking

then

* if app exited with okay, it is considered running

* if app exited with not okay, you get error on start

but obviously then app need to be of self-forking type.

It's an iffy problem to solve; I guess systemd could have some wait interval to wait for service that might crash some milliseconds after start but that would delay every start unnecesarily

Right, like I said I don't think any init system has actually figured this out but that is what as an admin I actually care about rather than startup times or the aesthetic properties of the configuration files. You would need some kind of semantics that define what it means for a system to be "ready". In practice we tended to lean towards some kind of polling of the service itself (which is at least slightly better than "sleep 3" or whatever).

Now, it's true that if applications would adopt systemd's Notify system this could really help, but this is equivalent to saying "if software were different it might be better".

Yeah its PITA problem.

Like, service can fork, return okay then die after a second coz someone set memory size wrong. No real sensible way for init system to prevent it.

Or takes 5 minutes to load then hits some bad data and crashes.

We just set services on auto-restart, monitor it and flag alert if service is restarting too often.

This requires the application to tell the service manager to wait for some signal that it has successfully "finished starting" beyond just "the process was created". There's no standard way to do it. In systemd's case this is done by opting in to inherit a socket that can be used to communicate that information (`Type=notify`, then calling `sd_notify()`).

Exactly: like I said below the pain point was never "generate a textual representation of the startup logic appropriate to the init system" but always "concretely define what you mean by a service being started or stopped and how to determine that".

there is one other way

If you set Type=forking

and the app exits with non-zero code on start then that will get propagated back to systemctl start

but notify is far better option even if requiring extra app support.

Fwiw nginx does do this correctly. I'm not sure what httpd does differently that it doesn't mimic this behavior.

People may not trust systemd because years ago there were some really bad defaults like https://github.com/systemd/systemd/issues/2402 that for instance mounted your firmware in read write mode by default.

That kind of thing leaves a bad taste in some peoples mouth. I am pretty sure some people have bricked hardware with systemd.

Or when they started killing user background processes (screen/tmux/nohup X) instead of sending SIGUP like POSIX/Unix has done since basically forever:

* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825941

As someone who runs mostly servers, I also find resolved annoying and have to rip it out. There's also all sorts of tomfoolery with /etc/resolv.conf in many modern distros, to the point I've taken to setting the immutable flag on it to prevent things diddling with it.

There was also the bug where systemD wouldn't allow usernames that started with a number. Poettering's (initial?) response was to tag it as "not-a-bug":

> Yes, as you found out "0day" is not a valid username. I wonder which tool permitted you to create it in the first place. Note that not permitting numeric first characters is done on purpose: to avoid ambiguities between numeric UID and textual user names.

* https://github.com/systemd/systemd/issues/6237

* https://news.ycombinator.com/item?id=14681377 (discussion at the time)

Nevermind that (a) POSIX allows it, and (b) so do the user manipulation tools of Poettering's employer at the time (RH).

I don't get it either. Systemd units are so much more powerful and flexible than init scripts. Writing those bash scripts for everything was a complete pain in the ass.

You may disagree with the architecture or design decisions made with systemd, but I don't see how it's possible to argue that SysVinit was in any way actually "better".

They're neither more powerful nor more flexible. Init-scripts can iterate & build whatever complexity they want. They can be far wilder. There's almost nothing constraining what an init script could be.

Systemd does establish easy-to-use & incredibly-powerful access a wide range of super-powerful knobs. And in a consistent & clear way, where-as each init-script would have a totally different way of doing config & slightly different approaches to the various hacks. And init scripts arent composable, where-as you can take your distro's regular unit, and drop-in some additional configuration easily, in a very eloquent way, across any bit of config. I think systemd is amazing & brings enormous clarity. It has enormous power.

But init-scripts still had more power. But it was reckless chaotic power that was enormously hard to tap, and required prodigal talent to develop these individual artisinal artefacts, which became, as a user, a thing of horror to rework. Init-scripts almost never were particularly flexible or powerful, but so help you if you ran into such an artefact.

Binary logs. Large attack surface. Ever growing features and pieces of the OS that it worms its way into. That's 3...

I never understood this binary log issue - it’s not like it’s some secret proprietary format..

Re large attack surface, anything prior was just security by obscurity, if not worse. You can’t secure chaos.

And on top of that: real seccomp sandboxes, dynamic users, chroot, isolated mounts, read-only mounts, isolated groups, process isolation, network isolation, virtual interfaces, virtual /proc and /dev mounts, capability boundaries etc etc ...all that stuff that is real painful to implement with OpenRC or other init script environments.

Out of the box, and can be used with every single systemd unit file.

Isn't that amazing?

Most of those sound like LXC features, and end users should be able to take advantage of them without having to be aware of how init might work on a particular distro.

The problem is that systemd tries to turn into something that does everything under the sun. It is not ideal and kind of far from the unix philosophy.

It’s almost like the Unix philosophy is bullshit at its extremes. System init is a fkin hard problem, you can’t just throw million small tools at it, the same way we don’t write web browsers by pipe-ing together curl and whatnot.

> System init is a fkin hard problem, you can’t just throw million small tools at it

You can't do it the way that everyone did it before systemd?

Sure, if I were to want a shitty, hacked together unstable system, where not even the few linux program could run on a different distro, as has been the case before systemd.

Do you think your description is accurate for Gentoo with OpenRC today?

Unix used to be about simplicity, and the composition of simple tools to achieve very productive outcomes. The shell was a part of that: learn it once, and you could do more-or-less anything with a Unix system. Including manage system startup and shutdown.

Linux has never really had that culture of finding the most concise but most general solution to a given problem. The results of that are everywhere, but systemd is a prime example.

To be fair, the world is not as simple as it was when Unix was designed either, and that's only partially because the Unix philosophy hasn't been widely adopted.

> Unix used to be about simplicity, and the composition of simple tools to achieve very productive outcomes

Then they should have consulted some CS books, as it doesn’t compose in the general case.

Also, how was using another init systems command different than using systemd’s?

I think it all comes back to people being upset about the Ubuntu PulseAudio switch. Ubuntu did a bad job when they switched to PulseAudio and caused lots of issues. Lennart Poettering was lead dev for both projects and it's weird how much the anti-systemd crowd get their panties in a bunch over him and also PA.

He fucked uo a lot in PA

He also fucked up some in systemd, journalctl being biggest example.

He also fucked up some in kernel, to the point Linus yelled at him

The critique is entirely deserved

> He also fucked up some in systemd, journalctl being biggest example.

"Let's just invent our own non-ACID file format instead using what's already available." (Like perhaps SQLite.) Howard Chu of OpenLDAP had a rant on this:

* https://www.linkedin.com/pulse/20140924071300-170035-why-you...

Does journald support off-host log shipping yet? Because for all its supposed replacing of syslog, as a sysadmin I still have to run (r)syslogd to send things to a central log host via RFC 3164/5424 anyway.

I'm pretty sure it's just him wanting to toy with databases and picking systemd to be his victim.

I'd expect when he finds new toy to keep himself busy someone else will fix it to actually work properly for the most time like it happened with PulseAudio

I also never understood the disdain for it. I feel like 15 minutes of reading about systemd keywords and I was on my way to making some very usable and reliable systemd scripts.

I don’t use sysv init anymore but I don’t feel like I ever fully understood those scripts.

I get the disdain. Beyond just reacting to change, systemv is fully scripted and thus a lot more accessible - one can read the source and understand what is happening fully once the kernel is done. On the other hand with systemd I feel we get this opaque magical complexity encroaching into previously fully grokkable user space and that is very hard to take, regardless of actual pros and cons.

I'm wondering how "writing unit files" even registers as a meaningful percentage of work (time) versus the parts where I (and others) see the problems. Looking at logs, having problems with journald, not being able to shutdown the laptop because something hangs.

Don't get me wrong, I've made my peace with systemd and I've never been an outright hater, but I did more sysadmin work in sysv and the transition times than now but I still get bitten by systemd more often per year. Usually small stuff, but apparently it only made those things better that never bothered me.

> has anyone who complains about systemd actually written a unit file vs an init script?

Yes, I've written quite a lot of both. My objection to systemd isn't really the init system replacement. It's OK (although I really don't prefer it over the System V style). My real objection to it is everything else about it.

> And all your logs are in journald

JournalD is an example of one of the "everything else" that I strongly dislike.

Haters gonna hate.

Some people just don't like it. That's their decision. In most cases its emotional.

Beyond complaints about journals being binary, which I get, there's usually not much more to the systemd hatred than "I just don't like it, humph (frown, arms crossed)".

It's an open source world..... the haters have the chance to come up with something that's better and if it is actually better then in the Linux merit driven ecosystem, then the better thing should win and be adopted, right?

Personally I love systemd - unbelievably useful and powerful.

Stallman used to say Linux should be called gnu/Linux because there's some gnu code in many distros.

More realistically it should be called Linux/systemd because systemd defines so much of how the operating system works.

They should have just offered a way to make it write text log files.

I have a prototype FUSE filesystem which renders the journal as a tree of .log files (https://github.com/awfulcooking/journalfs)

But my muscles finally learned `journalctl` and after reading the docs I mainly use it.

Patches welcome to make journalfs practical (i think it may be lagging output when you `tail -f` atm)

Would love to see systemd offer a virtual filesystem interface by default.

It's just as well. There isn't any software on my OpenSUSE Tumbleweed (rolling distro) machines that still only has sysv scripts. My Debian 11 router had some (nginx IIRC) but those went away when I updated it to Debian 12.

The more impactful change to me is the one mentioned a few lines down, about `systemctl --user` getting `CAP_WAKE_ALARM` so that user-privilege programs can set timers to wake up from suspend. As it mentions, GNOME Clocks has wanted to use systemd timers for a long time, and it's the de facto alarm program for Linux phones running GNOME / Phosh. https://gitlab.gnome.org/GNOME/gnome-clocks/-/merge_requests...

Not made by Microsoft, but 'embrace, extend, extinguish' which is a strategy of these folks I saw coming years ago, will no doubt surprise many...

I get that the real purpose of invoking "embrace, extend, extinguish" is to suggest shadiness and self-interested motives. Which, sure, you're allowed to suggest that. Maybe you're right, who knows.

But actually taking those words at face value...in what sense could systemd be described as "embracing" or "extending" any init system that came before it? Wasn't it always a from-the-ground-up different thing?

Is systemd solely an init system? Did it extend its scope far beyond that?

It's been openly and deliberately much more than an init system for the decade or so that I've been using Linux.

Myself I first installed Linux in 1994, and there are many people on HN that were using it and other Unix systems well before that. It might be that the time of "when you started using Linux" plays into views of systemd ...

Yes... Taking over the lucrative business of making the init system. That must be their goal ; ) There's really no extinguish here - other init systems still exist for those who have reasons to use it.

Yeah I think we all saw it coming. At least kdbus didn't happen.

> Not made by Microsoft

Poettering has been a Microsoft employee for like a year now.

Instead of writing init scripts (that are far too often done wrong compared to what some lunatics say "ITS SIMPLE", even by big software packages) you just put like 5-10 lines in a text file (hell, aside from boilerplate/info about service you need like... one in many cases)

How's that "extinguising" anything?

It's not, but they also never "embraced" the old init scripts either, they just supported it for backward compatibility.

That said, there are a lot of legitimate grievances out there against Systemd. But it has little to do with using unit files instead of scripts.

yeah like journalctl and especially its unfortunate useless binary format

… you’re mourning SysV init scripts? why?

I mourn it. I've been typing '/etc/init.d/apache2 restart' for about twenty years now. Why change it?

Would it really be that hard to have a symlink from /etc/init.d/apache2 to /bin/systemctl and teach systemctl to do the right thing?

There's a generic wrapper which existed for many years "service apache2 restart" will work regardless of the init type.

You can also put "systemctl $1 apache2" in /etc/init.d/apache2 and keep using it. It's really not a big issue.

Be the change you want to see.

If nothing else, sysv scripts were conceptually really simple to the point that you look at it and think "oh, it's just a shell script; I could actually just sit down and reimplement the whole system in a weekend". In fact, much of my personal dislike of systemd is that it started getting popular right as I was having that revelation and got to watch a pile of shell scripts that I could just pop open in my editor get replaced with an all-encompassing black box (yes, I know it's open source, but "you could go find the source and try to read C" is not the same as "vi /etc/init.d/foo".

That's a lie

https://news.ycombinator.com/item?id=36513889#36514362

> In fact, much of my personal dislike of systemd is that it started getting popular right as I was having that revelation and got to watch a pile of shell scripts that I could just pop open in my editor get replaced with an all-encompassing black box (yes, I know it's open source, but "you could go find the source and try to read C" is not the same as "vi /etc/init.d/foo".

That theoretically is a disadvantage but practically ~90% of the systemd scripts are "point at binary, pick user, group and work dir" and entirely trivial to understand; distro ones often are a bit fancier with limiting app capabilities but those are far harder to do in init script in the first place.

For vast majority of apps it's literally just 'here is binary, run it'

And more important part is that I DON'T NEED TO EDIT ORIGINAL if I just want to add something.

I can take 100% stock distro .service file, add

    Restart=always
    RestartSec=60s

and now any time app dies it will auto-restart.

Or add proxy environment, or anything else, via just override files, no need to litter our repositories with modified init scripts that just have like 3 lines changed.

> That's a lie

Oh, well now that you've called me a liar I suppose I have no choice but to agree with you. (/s)

Here, let's have a worked example!

https://git.alpinelinux.org/aports/tree/main/nginx/nginx.ini...

In that entire file, there is exactly one function that exceeds 3 lines; the "upgrade" command, which AFAIK doesn't exist in systemd-land.

> That theoretically is a disadvantage but practically ~90% of the systemd scripts are "point at binary, pick user, group and work dir" and entirely trivial to understand; distro ones often are a bit fancier with limiting app capabilities but those are far harder to do in init script in the first place.

Partially true; I'll happily agree that 90% of daemons are the simple case. Of course, that's true regardless of the service manager in play; 90% of systemd service files are trivial, and 90% of initscripts are trivial. I'll also agree that there's an advantage to making the easy stuff common; if I were actually writing initscripts for a distro I'd have them start by including a library (probably `. /etc/init.d/_common.sh` or something like that) and then most of them would also just define the ~3 needed variables and use the default function implementations. And it would still be easier to follow than a compiled version.

But you aren't using bash!

    #!/sbin/openrc-run

> openrc-run is basically an interpreter for shell scripts which provides an easy interface to the often complex system commands and daemons. When a service runs a command it first loads its multiplexed configuration file, then its master configuration file, then /etc/rc.conf and finally the script itself. At this point openrc-run then runs the command given.

It essentially recreated "init script specific DSL" that systemd unit files are!

So you don't have "conceptually simple" SysV script, you have massive wrapper that takes care of the ugly edge cases around starting and managing services and is also is entirely dependent on openrc stuff to work.

And that's not even end of it!

    start-stop-daemon --signal QUIT --pidfile $pidfile.oldbin

that's the wrapper Debian and other distros used a lot

So you're using openrc wrapper

to call shell

that calls another wrapper.

To run a service

Systemd equivalent:

    [Service]
    Type=forking
    PIDFile=/run/nginx.pid
    ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
    ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
    ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
    ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
    TimeoutStopSec=5
    KillMode=mixed

I rest my case

A unit file that wraps start-stop-daemon, has type=forking, and uses PID files.

If it weren't a silly one-off on Hacker News, and if I hadn't already pointed out this foolishness years ago with exactly this service, it would be a prime candidate for the systemd House of Horror.

* https://unix.stackexchange.com/a/504374/5132

* https://jdebp.uk/FGA/systemd-house-of-horror/

openrc-run is itself merely a shell script, by the way.

Yeah nginx one could just use systemd itself to send sigquit. I'm guessing either author didn't read up on all the options or the option to customize signal in systemd was added later

Because it will one day be “invented” again as a “scriptable init”

Can we stop these fking stupid takes? It’s like in every wayland and systemd thread..

In this case it was used as an instrument of good. The strategy isn't inherently bad, it is how it is used and using it to kill Sys-V init was a very good thing.

Also they are ending support for unmerged-usr (/usr/bin and /bin as different directories). On one hand, here's Redhat (and Pottering) using their position as dominate service manager to bully distros and software into adopting their views of the world. On the other hand, systemd has been a great step away from fragmentation in distro-land.

Are there any major distros (or even minor distros) that haven't committed to the /usr/-merge at this point? I know Debian has been having issues, but for the most part that decision has been made.

Debian didn't really have any issues with it either, merged /usr is the default since two stable versions ago already. There were just some troubles with internal politics around the topic.

Debian has merged /usr too. At least my current Debian 12 install does, and I'm pretty sure it had it before I upgraded it from Debian 11. I believe Debian's problem is (was?) only with migrating existing pre-merge installs.

The only "problem" Debian had was that dpkg maintainer had differing opinions on how the merge should have been done and used dpkg as a vessel to express those opinions. Merged /usr was already available in stretch, is the default since buster, and support for non-merged /usr has been dropped in bookworm (all non-merged systems are converted on upgrade now).

Works fine. Debian 12:

    lrwxrwxrwx 1 root root 7 04-28 23:21 bin -> usr/bin
    lrwxrwxrwx 1 root root 7 04-28 23:21 lib -> usr/lib
    lrwxrwxrwx 1 root root 8 04-28 23:21 sbin -> usr/sbin

that on install that was installed somewhere in 2007 then just dist-upgraded for 16 years.

The cool thing about Linux is you can start your own distro where like minded people can contribute.

> On the other hand, systemd has been a great step away from fragmentation in distro-land.

Why are you making it sound like a monoculture is a good thing?

There's plenty of room between severe fragmentation and total monoculture.

> step away from fragmentation

Not really, it has increased the number of people using alternative init systems like OpenRC, Runit, S6, etc.

Before SystemD, SysV was even more mainstream than SystemD is today.

That does not match my memory. I remember it being very split between Upstart and SysV. Now the major distros have pretty much all standardized on systemd.

Nor even was sysvinit always set up consistently, e.g. some distros just had init.d and rc.d, some had all the runlevels, and what those runlevels meant were different distro to distro

But Gentoo switched to OpenRC, Debian forked to Devuan, Arch forked to Artix: all these main distros were using SysV and now a part of their users are using an alternative system.

What's the market share of those though? I've networked with a lot of devops/sysadmins and I can't to my recollection ever remember somebody using Devuan or Artix, and it's rare indeed to hear somebody using Gentoo.

I'm not dismissing them because they aren't popular, but I do think it's not very reasonable to call the market fragmented when the nons are such a small percentage of overall use.

Who is ever going to run Arch on a server anyway?

I've used Obarun a bit in a VM and enjoyed using 66 Suite. I wouldn't use it at work (again mostly because it's Arch and we use Docker for everything anyway so no need for init), but I would consider using it on a home project server.

Yeah ain't nobody using those.

GNU/Linux is a nebula of many blocks that sometimes compete for the same role, but heavily rely on intercompatibility and adapter layers in order to allow users to choose the one they prefer in each spot (I know this has been less the case with Gnome recently, for example, but the ideal would be to stick to the principle).

Even though the SysV shim layer can be maintained outside of RedHat, their decision is still a spoke in our wheels.

Right but the decision has been made and ones that didn't go with systemd are probably fraction of a percent by installation count. Doesn't make sense to keep the cruft, especially that the alternative (writing unit to some app not delivering one) is like 5-10 lines of text and 5 minutes of work.

still using init scripts on OpenRC.

> Now the major distros have pretty much all standardized on systemd.

Which is why I need to stop using Linux.

That's a bold strategy cotton, let's see how it plays out for them. ;-)

But seriously, I expect to see a lot of software lose the ability to be run on systemd because they don't ship a service file.

As far as I know, plugging into init is the responsibility of the person packaging the software for a particular distribution. If a package for your distro is not available and you're just compiling from source or downloading a standalone executable, you have to write your own anyways. It might be different if I were an experienced bash scripting guy, but I find chucking the path to the thing an "ExecStart" line a lot more straightforward then writing a script that parses arguments and calls start-stop-daemon and all that.

My fear is that software will stop shipping with an init script option because of this. I really need to stop using Linux sooner rather than later.

The BSDs (Free,Net,Open, and Dragonfly) are excellent if you’re looking for an open source *nix that isn’t Linux.

I am specifically working more with FreeBSD as I think the day is coming that Linux won't have the ease of configuration it does now.

Yes, I've been planning to move all my machines to BSD for a long while now. I've been procrastinating because I have a lot of machines and the scale of the task is a bit intimidating. The window for procrastination appears to be ending, though.

You could jump to Gentoo + OpenRC

Or have a look at all the distros that don't use SystemD!

My fear is that more and more Linux software will depend on SystemD facilities directly.

I've used Linux for a couple of decades or so. SysV initscripts have been a pain for many of those years. Do you do RH/Mandrake/Mandriva flavoured or the SuSE ones and other oddities. I decided to wander over to Gentoo for my personal stuff for a decade or so. Now the bloody initscripts need to conform to OpenRC. At least Mr Marples gave me some help with this.

A systemd unit is a simple thing to rustle up. I have recently been informed that an OpenRC script is too these days and I'll fiddle up a VM and have another bash with Gentoo. My job as Managing Director of my company is to maintain an open mind!

I don't think that not shipping a systemd unit will crap out any piece of software. I've seen quite a few packages of code, source and that and an initscript in ./contrib probably works on Debian and farts loudly on SuSE or vv. Who cares? systemd can run it quite easily and with minimal pissing around. With logging and all the other useful stuff.

A trained monkey can write a service file. There is no software that can start with a SysV script that isn’t easily ported to a service file.

I assume someone will introduce a compatibility shim, either at the systemd layer itself, or by running the software in a container.

They could just fork the sysv-generator and continue maintaining it.

(Generators are arbitrary programs that generate systemd units at runtime dynamically. Eg fstab-generator parses /etc/fstab and generates .mount units. The sysv-generator generates a .service unit for each sysv init script it finds, and is what is being removed here. https://github.com/systemd/systemd/blob/main/src/sysv-genera... )

Of course, that begs the question of why it's being removed in the first place; if it's really that modular, it shouldn't be any effort to maintain.

The answer is they don't want to be ones maintaining it; after majority of distros migrated it's just a wasted effort on their part.

Distros that use it can maintain it for their needs; it's pretty logical.

There is very little reason for distro to support systemd and support sysv scripts.

I am surprised it comes from red hat to be honest, if anything I'd imagine them needing it for some legacy software their customers use that might come only with init scripts.

Then again, in most cases rewriting init script to run as systemd unit is very straightforward; but then I saw some fucked up init scripts too, like ones trying to start and manage multiple services at once (here hearthy "fuck you" to packetfence developer that thought that would be a good idea)

> it's just a wasted effort on their part

I guess my naive expectation is that it should be any effort, hence the question.

> Then again, in most cases rewriting init script to run as systemd unit is very straightforward; but then I saw some fucked up init scripts too, like ones trying to start and manage multiple services at once (here hearthy "fuck you" to packetfence developer that thought that would be a good idea)

LOL, now you've triggered an old memory of IBM WebSphere MQ and its... charming init scripts. It was almost easy to replace them with service files, except that IIRC template unit files can only use one variable to template and we needed two, so I did something ugly. (I don't even remember what, just the gross feeling.)

one variable is the thing after @, IIRC called "instance"

but you can use that variable to specify EnvironmentFile that has more variables :)

and it also neatly get per-instance config into separate file

or use configuration management to generate per-service files instead of using instances

> but you can use that variable to specify EnvironmentFile that has more variables :)

I mean, you could, but when you really only had 2 variables that feels pretty kludgy.

> or use configuration management to generate per-service files instead of using instances

I suspect that's what I actually did, since ansible/chef/puppet/whatever is happy to nest for loops and template the result. It was just annoying that I couldn't do it in place.

well, instances weren't really done to customize variables, their design was pretty much to do "a program that can be given a different config file as a parameter", like it is done for say postgresql instances.

I'd even dare say: it'll be safer in other hands.

It’s trivial to execute an existing script from a SystemD service. Like, you just tell the service to run the script.

This is so much nothing it can’t get nothinger.

Writing a service file is trivial though

Updating software that hasn't been updated in a long time and so far worked just fine is not trivial though.

How is a SysV init script different than systemd starting that exact same script on the same system?

sysv init scripts are more complicated than just a list of commands to run to start the service. They have start / stop / reload functions and express dependencies. The sysv generator handles all of those. Like this jank: https://github.com/systemd/systemd/blob/08423f6d30f5db045b8a...

So it's not as simple as "Just write a .service with ExecStart=/etc/init.d/whatever start", but it's also not too complicated, especially if someone forks the sysv generator and keeps it working like I said in another comment.

Well, wait: sysv scripts can express dependenices but they don't have to. The admin is totally free to just set the start and stop orders in the runlevels themselves (that's how the BSD world does it). Setting up a dependency system means you're just making a worse systemd and I've never really gotten the appeal of it.

Fun fact: Debian, thanks to those dependencies, had ability to start SysV services in parallel, before any of that fancy upstart/systemd happened.

It just had batch job that solved the dependencies, ordered the init scripts numerically, and then started in batches all of those that started in same number (IIRC, there might've been a bit more fancy stuff there too).

It was kind of worse systemd but systemd wasn't around back then

And I had a staticly compiled make that read /etc/rc.mk in my toy distribution in 1999; none of this is a groundbreaking idea. There were plenty of ways to do parallel startup in the oughts but admins rejected them because parallel startup is largely a bad idea. Start the services in the order I tell you to, one at a time, logging the results. Anything else will eventually bite you.

Well, since the 90's we had more than one core in the average machine.

There is no reason parallel start would be problematic; I even got stuff working that needed network access to get creds for encrypted partition that was required for database to start and it was handled flawlessly.

More than that, if for some reason that file system wasn't mounted it would be when I told systemd to start database, it would run the service retrieving the secrets, then the mount using it, then the DB.

The worse story is with shutdown; by default most distros will cut ssh access before services are closed so if some of them hang you can't debug that remotely. But that's mostly bad settings rather than bad design of the daemon

I think you're definitely right that shutdown is the harder problem and it's one that init systems in general pay a lot less attention to, unfortunately. If the system has a problem coming up it often at least gets into a state where I can access it; if it hangs after being told to reboot, well... I once had to drive from Arlington VA to Seacaucus NJ to press a power button on a rack server in our datacenter, and I don't like that.

You couldn't use the DC's remote hands?

IIRC the issue was if the remote hands opened our closet they would see we were over our contract limit for servers in the rack. The late 90s were kind of the wild west when it came to colos.

lol, yeah, been there.

you could, most modern servers also have out of band controls, but it is still PITA because you don't know whether something is wrong or it just takes extra few minutes for whatever reason

It's generally not; I've never understood the claim that either one is particularly "complicated". You either write a set of sh commands that start the service the way you want to or a unit file that has that shell command plus some random metadata crap about it. Neither of these options are particularly difficult and I've literally never understood why people want to die on either side of this hill.

The pain point was never "generating a textual representation appropriate to the init system"; the problem was always the underlying startup and teardown of the service itself.

If the startup script "hasn't been updated in a long time" and you're scared of breaking it, why would you update the operating system to whatever FUTURE version actually drops support for SysV scripts?

Business requirements are never "just don't update this machine", you often have conflicting requirements where one piece of software has one set of requirements and another one has different ones. You often need to update systems to stay compliant with security requirements and those updates are generally non-negotiable.

I'm a fairly cautious sysadmin but I can't say that replacing an init a script with a systemd service is something I'd be afraid of doing

DO you have actual experience in having problems converting init scripts to systemd or you're talking entirely out of your arse?

From my experience yes, even the legacy stuff is easy enough to convert.

The compatibility layer had the benefit of letting you write your service once for both init systems.

So they cease to exist for linuxes? Because there is no sizeable init system besides systemd.

Lennart Poettering is now working for Microsoft.

is he still working on systemd?

He made the commit that added this deprecation notice

https://github.com/systemd/systemd/commit/7474097d51907c5cc1...

Two things that are sure to set fire to the comment section: a negative article about Apple, and any article about systemd.

Good luck out there, friends.

Don't forget Wayland!

$ ls -al /sbin/init

-rwxr-xr-x 1 root root 52400 Apr 3 02:25 /sbin/init

$ cat /etc/debian_version

12.0

running:

postgresql, postfix, dovecot, clamav, nginx, chrony, clamav, isc-dhcpd, bind...

This is an amazing release. As far as controversy clickbait topics go, personally I'm much sadder to see umerged-usr bite the dust, even though i'm a huge fan of merged-usr, just because it was super nice being able to boot off whatever / you have and drop another systems' /usr in. But at least there's systemd-sysext to allow some combining overlays. Anyhow, excited to talk through various changes that look exceptionally neat to me:

RootEphemeral=, which lets processes run in a filesystem snapshot; super neat for seeing what changes a process might make to your system. https://github.com/binpash/try was a popular submission a couple days ago, & this builds in a similar capability to the os.

RestartSteps= and RestartMaxDelaySec= to finally finally offer some level of control over exponential-backoff in the restart behavior.

FileDescriptorStorePreserve= is kind of mysterious, lets systemd store open file descriptors for a process even after it terminates. Some interaction with systemd-analyze. Not sure what this is for exactly but definitely cool extra visibility/observability of processes. I tend to think it also potentially could be useful allow for some weird/fun stuff like restarting services without closing ports.

Soft-reboot which can be useful for switching roots without having to juggle bootloading. Very useful for anyone doing partitioning or what-not. And as I hinted above, you can pass file-descriptors, meaning you can avoid service-interruption while rebooting; what a neat trick.

DelegateSubgroup= puts processes inside a sub-cgroup inside the top level cgroup. So as a user you could have a bunch of sub-cgroups that you can manage, letting you juggle/manage resources within your cgroup. This sounds a lot like Dixon's incredible talk technique of cgroup management, Linux memory management at scale. https://www.youtube.com/watch?v=beefUhRH5lU&t=2241s This is also easily what I think the worst most pitiful saddest part of the Kubernetes story is: a complete unwillingness to let Linux manage memory & use cgroups well, in favor of it's own scheduler which makes all the decisions.

iocost calculations for io devices, to configure/tune io/qos cost for devices. has a huge existing db of drives. https://github.com/iocost-benchmark/iocost-benchmarks

systemd-sysext gains a systemd-confext twin. the former manages overlays for /usr and /opt, the latter manages overlays for /etc. Very nice portable services tool now. Maybe the capstone work completing Revisiting how we put together Linux Systems. https://0pointer.net/blog/revisiting-how-we-put-together-lin... .

Using efi vars to save the hibernation partition, which should be a huge win for hibernation working everywhere.

Some other neat bits:

CAP_WAKE_ALARM rights to user sessions, for setting long-running timers. Sigqueue support to send processes signals with associate values. list-paths verb to show paths. systemd.tty.* kernel arguments for setting up ttys. Early load virtio_console module if it detects it's runnin in a VM to get console up earlier. Upholds= gets a new .upholds/ drop-in directory for keeping other services running (not just starting them like Wants=). sd_journal_get_seqnum api call so apps can have mechanistic sympathy with journald. Ton of bootloader, boot-security, repartitioning, disk-encryption enhancements across the map. SetTTY to for updating the TTY of a session; so useful for ssh'ing in.

udev creates /dev/loop/by-ref/* files with helpful names. systemd-resolved gets a StateRetentionSec= to let you cache & use old records if a nameserver isn't responding. Most systemd services now have some standard signals they accept to tweak config, ex: changing log level, log target, trim memory, print debug mem info. machinectl & networkctl gain "edit" and "cat" capabilities for viewing config. systemd-firstboot --reset which is hella useful for cloning machines & updating machine-id. Services now sd_notify with their EXIT_STATUS on exit; part of trend of making services more observable/operable everywhere. new systemd.mount-extra= kernel arg to ask for mounts at startup.

I love reading release notes in general, but this was absurdly delightful. It felt like in the past sometimes systemd was focusing on new feature growth & ideas, but I'm seeing huge maturation & growing in in this release. Really leaning hard into everything alive & responsive on via signals and passing around file-descriptors is excellent excellent excellent systems-wonkery. The fit and finish here is going way up. There's just a ton of damned-useful ergonomic wins that make everything super easy to find & see & touch & manage. 254 is an epic release.

Reply on news.ycombinator.com