Systemd's slowly but steadily increased idealism about the world | Hacker News Reader

Systemd's slowly but steadily increased idealism about the world

utcc.utoronto.ca

85 pointssynthmeat58 comments

Good? I have gripes about systemd, but this is not one of them. Indeed, cleaning up the multi-decade Gordian knot of distro-specific crappy Sysvinit scripts was the entire point, as they made clear right from the beginning. If systemd compromised a little on that in the early years, that was probably just to get a foot in the door.

> These dependencies are predictably wrong in any number of cases, because they've never been really used before.

Then fix them. Isn't that considered good software engineering in all other contexts? At least where I work, you're not allowed to merge until all the unit tests are green, and if someone writes a new unit test that reveals your component was broken all along, you fix it.

The comparison to security is not valid at all. I was on the side of pragmatism and the letter-writers, and against the Guardian and the purists, on the recent WhatsApp fracas, which you'd think would pit me against systemd here, but the analogy is flawed. We have to be pragmatic about security because it involves the choices of uninformed and uninterested users, and we have to maximize their security given that they frequently make bad choices and we have little hope of making that better. systemd is exposing problems that were created by developers and can be fixed by developers.

If I have code that is:

  executeStep1();
  //Step 2 requires step 3 to be run first
  executeStep2();
  executeStep3();

I don't expect my compiler to go ahead and run step 3 first. Systemd is choosing code comments over code organization. The tests are green, until we run them under a compiler than prefers comments over procedural organization.

I don't mean to advocate for systemd's approach, but they are not the first compiler to give meaning to information in comments.

For example, there exists a Javascript compiler that puts static type annotations in the comments: https://github.com/google/closure-compiler/wiki/Annotating-J...

For static analysis and linting in C information gets held in comments as well.

Unfortunatley, about the only thing everyone can agree on is:

  /* fallthrough */

For when you intentionally don't put a break in a case statement.

> I don't expect my compiler to go ahead and run step 3 first.

You would be surprised[0]. In low level code (in some cases) you'll need memory barriers[1] to ensure the proper order of instructions execution.

Even in higher level code your compiler's optimization algorithm may decide that it's faster to execute step 3 first if it thinks the result would be the same.

[0] https://en.wikipedia.org/wiki/Out-of-order_execution

[1] http://irl.cs.ucla.edu/~yingdi/web/paperreading/whymb.2010.0...

Out-of-order execution always has the same answer that would occur from a single threaded perspective. There's no "thinks the result would be the same" about it - the result is the same.

You get into trouble when you:

1) assuming ordering to external peripherals (e.g., memory mapped I/O)

2) assuming stores/loads are happening in a specific order (e.g., other threads either pre-empt or are simultaneously running) causing data races

Your response makes it sound like a compiler could reorder functions like:

  x=5;
  x=add(x,5);
  x=multiply(x,10);
  printf("%d", x);

and you could expect to get either 100 or 55 as an answer if you didn't use fences. That's not true. The answer will always be right (100) no matter what OOO, speculative execution, etc happen.

If you read my post carefully the "thinks the result would be the same" was about compiler optimization case, not processors' out-of-order execution.

Take this example:

    a = 5;
    b = sqrt(44);
    return a * b;

If the optimizer is sure (and strict aliasing[0] also helps to be sure in many other not so obvious cases) that the sqrt function is not using the a variable then it can swap lines 1 and 2 in the generated machine code, and the result will be the same.

In your example changing lines 2 and 3 will alter the result naturally so the optimizer won't touch it.

[0] http://dbp-consulting.com/tutorials/StrictAliasing.html

I thought the headers were interpreted by sysvinit as well, as they were intentionally part of the design. Am I mistaken?

System 5 rc (rc, not init, note) didn't have such a concept. Even the original van Smoorenburg clone of rc didn't have this.

The LSB headers came along some years later. They are one of two different header styles that systemd's conversion tool understands, in addition to supporting the case where there are no headers at all. There are multiple extensions to van Smoorenburg rc. The LSB headers are just one such extension, understood by toolsets on some operating systems and not others.

* http://unix.stackexchange.com/a/233581/5132

Because init scripts are being used in such a strange and non-rigorous way (configuration and operation combined in single files, with execution order implied by the name of each file), I don't think it's fair to draw such a direct parallel to the more formal languages used today for programming.

The fact that they are syntactically formatted as comments is irrelevant. There is a spec, and that spec is respected by many tools besides systemd, including insserv (which was used for parallel boot on Debian for quite a while before systemd became default).

Because shell scripts are such a terrible format for init scripts, they were standardized as comments to the shell. But they are not comments simply for human use.

An analogy is the type-checking syntax in Python 2, which couldn't pick up a syntax change like Python 3 does. They're formatted like comments. They're not arbitrary strings for amusement, and a type checker is entirely right to complain if the appropriately-formatted comments are invalid. If you want to express things for humans, you're welcome to use normal unstructured comments.

> we will assume that declared dependencies are correct, gain a minor speed boost, and it things blow up it's not our fault'.

Wait that actually sounds pretty reasonable. What's wrong with systemd taking that position exactly?

Nothing, it's just it broke something this guy maintains/owns because his scripts where literally broken. And now that means Systemd is some horrible monster because it executes exactly what it is told to, and doesn't double guess the input because of some legacy crap.

Garbage in, garbage out folks.

His scripts were not broken, his LSB headers were, which would be obvious to anyone considering they're script comments which no tool had ever executed, which means they're 15 years of bitrot, I've seen examples in documentation bitrot in under 6 months.

The "garbage in" part is entirely systemd's fault, it was their decision to make the system's boot process rely unconditionally on LSB dependencies which were all but guaranteed to be broken.

I'm not defending systemd here, but in general, if your files have standardized headers that indicate the wrong information, how is that anyone else's problem?

Seeing that no one is apparently using the information until systemd decided to, it seems like systemd being pedantic and the problem.

And while i can't pull another example out of a hat right now, i am sure i have seen other examples of systemd being pedantic about information nobody else have relied on before, and in the process getting its panties in a twist.

Again, not defending systemd, but if you have a structured header in your file which is following a standard (even if no one is using your header or that standard yet), and your header is wrong, is it really anyone else's fault but yours?

Shouldn't the fix simply be to remove the header (if you don't want anyone using it) or fixing it (if you do)?

Why ask tools, which interpret the header according to the standard, to do something different because your header is wrong?

And once more we are myopically focusing on the technical when the root of the issue is historical and social (if not organizational).

While the headers have existed, they have historically been underutilized and thus poorly maintained. Thus a newcomer can't just walk in and suddenly make them massively important while claiming the breakages that come out of it is not their problem.

> While the headers have existed, they have historically been underutilized and thus poorly maintained.

They are human-readable, so even if no tool used them, they still have an effect. Someone might read the header and incorrectly move the service in their sequential boot order, for example.

Would you recommend that the person reading the header also "fix themselves" instead of fixing the header or filing a bug against whomever owns the incorrect file?

> Thus a newcomer can't just walk in and suddenly make them massively important while claiming the breakages that come out of it is not their problem.

Someone decided to put the two things together, right? Someone combined this "newcomer" and these old files with the bad headers? The onus should be on that person to make things right, not the tool which is using the headers correctly. It could be a distribution, for example, but it isn't the tool.

Update them or remove them if they are wrong. Why campaign to leave them intact and incorrect?

At least Debian and Ubuntu have been paying attention to LSB headers for years.

They may be comments to the shell, but they're input data to insserv.

Broken being the important word here.

Also after a cursory google this[1] SO posts appears, which says that the headers where most certainly not ignored.

1. http://unix.stackexchange.com/a/233581

Actually the dependencies are considered by several tools, like chkconfig (source: have actually done this)

But yeah, having configuration in a comment is as absurd as it looks

Bitrot has to be fixed. Think of systemd as your friendly LSB header linter.

> Think of systemd as your friendly LSB header linter.

A linter does not break the system it's running on.

It's just an extreme form of error reporting.

This is a time-tested way to make people fix broken things that work by accident.

What's wrong with systemd taking that position exactly?

They are gathering "dependencies" from script comments which have never been executed in any other system.

Equivalent to:

  //#include "myfile.h"

That is factually incorrect. The insserv software makes use of these dependency declarations for dependency based parallel boot sequencing, which has been in use by Debian since 2005 (based on the comments in the changelog), and I believe it's been the default in Debian since 2008.

What's wrong with that:

1. They aren't really declared dependencies, they only look like that.

2. nothing else, once you understand point 1.

https://xkcd.com/1172/

Whenever this one comes up, i am unsure how to interpret it in context. Because i am tempted to side with longtimeuser4 on the issue, but not to the point of re-enabling CPU heating.

Instead, implementing some way for holding a user configured key for a user configured period of time to trigger the wanted action. Thus both workflow and fix is preserved.

As a former linux&bsd localhost admin and a seasoned programmer, I cannot understand all the fuss about systemd and init procedure in general. Is it really that hard to start and/or stop programs with arguments, given their dependency graph declared somewhere? Why do we need sqlite built in there? Why all that hardcore coding process involving furious coders? Seems like I'm doing things like that around every two months while solving yet another graph-related business problem.

Why such drama, anyone?

I'm not up to scratch on the history of this, but systemd is coming from the Freedesktop.org camp, which also game us GNOME 3 aka Unity. This group seems to have the biggest case of ivory-tower syndrome in all of OSS. :(

The infuriating part is that the whole thing is backed by Red Hat and very closely followed by Ubuntu. So basically you have emotionally immature "devewopers" with overinflated egos being given the keys to the Linux kingdom.

They've even made the kernel devs mad, but there's very little the kernel folks can do except shake their heads in stunned disbelief and repeatedly outline the concrete examples of where things have gone wrong.

I'm saddened to say this, but it looks like Linux has gotten so popular that its collective competency has taken a nosedive.

The problem is political. systemd is being looked at by enough people that issues like the one raised in this post eventually come to light. It's what happens next - the social/communicative stage - that is where the breakdown happens.

I honestly don't get why things are like this.

At this stage, I'm in the "systemd is in practice fairly bad, but there's not very much I can do about it" camp. It now seems unfortunate that its ideological opponents have behaved so poorly in so many cases; I'm pretty sure this drastically colored community evaluations of the technical merits of their arguments.

My hope has been that systemd will, over time, increase enough in quality and settle sufficiently into its existing scope to recede into the quiet background of the system administration experience. (Where init systems have for the most part lived while I've been paying attention.) Perspectives like the article here leave me kind of uneasy.

"behaved poorly"? Care to give an example?

Given the importance of Systemd, its better to follow idealistic principles. Pragmatism in central components leads to technical debt that is hard to remove because it would break all the lazy components that took the "pragmatic" (lazy) route.

On the one hand, we talk about bad software, too many bugs, bad architecture, messy init scripts. On the other hand, we are too lazy to follow idealistic principles that improve overall software quality.

I recently ran into a similar case of idealism: in this case, systemd assuming that a laptop's lid "button" (and its S3 state) actually works correctly and assuming that the user wants the laptop to go to sleep if the lid is closed (regardless of power management settings).

For context, the laptop is an old one running openSUSE 13.2. User hands me the laptop to take care of routine updates. All of a sudden the screen goes blank, even in non-graphical modes. I was eventually able to read through 'dmesg' long enough to find out that the system was being told to go into standby exactly three minutes after boot.

After some WWW searching, I found out that systemd (or maybe logind specifically) does this as a "safety measure" (so that your laptop doesn't burn up in your backpack). Never mind that there are plenty of legitimate reasons for closing one's laptop lid besides putting it to sleep (for example, plugging into an external display). This wouldn't have normally been an issue if it weren't for the incorrect detection of the lid state; despite the laptop lid being wide open, it was mistakenly detected as closed. Further, the laptop has never really cooperated with Linux when it comes to standby, so instead of going to sleep, it'd end up just hanging at a blank screen entirely unresponsive, requiring a hard reboot.

Like, I appreciate that systemd wants to have my back, but I feel like they're overthinking things. I was thankfully able to edit some config file to tell systemd to ignore laptop lid events, but it's still rather silly.

That's really strange. I've run multiple laptops in clamshell mode, with the system on my desk, closed, with external devices for all human interface. Do they not support that workflow, or are they being smart, and only doing that when there are no external displays connected?

From what I gathered, they're trying to be "smart" about it by detecting if there are "docks", but I didn't really investigate much further (in particular, if "dock" means "external monitor is connected") beyond setting the relevant action to "ignore" in some config file (don't have the laptop in front of me at the moment, and my own machines don't use systemd, so hard to be more specific, unfortunately).

Some lid switches are broken:

https://who-t.blogspot.co.uk/2017/02/libinput-and-lid-switch...

Systemd is in the "extinguish" phase of its dominance. "Not my problem" is something you can only say once you're on top.

The extinguish phase will be when the submarine patents surface and you can either pay up or shut down or go to court.

In the projects I've used systemd in, it's been pretty smooth though most of my daemons are pretty straight-forward. As others have pointed out most of the issues the author seems to be running into are dev issues and not the fault of systemd itself.

I'll take systemd over supervisord any day.

This conflict between idealism and pragmatism makes me think of the difference between Google and Microsoft's engineering cultures.

The attitude towards backward compatibility is a major difference between the engineering cultures at Google and Microsoft. Google relentlessly deprecates APIs while Microsoft supports things forever, warts and all.

Google's APIs are very idealistic. For example, Google Guice is a very idealistic dependency injection framework using and abusing the Java type system while Microsoft's APIs are very pragmatic and some are only slightly object oriented.

This is how I've largely seen the divide between these two companies as well. There's pros and cons to both approaches.

Personally I favor Google's approach to Microsoft's as I am not a fan of technical debt, but if I were an enterprise customer I might feel differently.

As a developer, I much prefer Microsoft's approach. That way I don't have APIs pulled out, products being abandoned and services axed whenever Google feels like it.

And let's not get started on the absolutely awful quality of APIs Google puts out.

> If you're not solving problems in the real world, you're not really solving problems; you are just being smug and 'clever'.

systemd is a project that wants to change what the real world is. It makes no secret of this. See, for instance, the very small /etc/os-release file: http://0pointer.de/blog/projects/os-release.html

"It's our hope that by dropping support for non-compliant distributions we gently put some pressure on the remaining hold-outs to adopt this scheme as well."

Sometimes they aim too high and it just doesn't work (see, e.g., the stateless systems stuff that assumes that everyone is bought in completely and also loves btrfs). When that happens, they don't start dropping support for everyone, they just quietly give up. But I don't think that support for SysV init scripts or /etc/fstab is evidence of a changed view on pragmatism. The way to change the world was to start with compatibility the world as it is, and push steadily for improvements. Device naming is unambiguously a place where the world needs to get better, and systemd is right to push for improvements (although I may disagree quite a bit with what specific improvements they're pushing for and how!).

Also note that Poettering is quite specifically is okay with /etc/fstab because he doesn't see anything he wants to change with them: the format is expressive enough for everything he envisions. https://github.com/systemd/systemd/issues/4961

A single format in a well defined place is better than /etc/redhat-release, /etc/SuSE-release, /etc/debian_version, /etc/arch-release, /etc/gentoo-release, /etc/slackware-version, /etc/frugalware-release, /etc/altlinux-release, /etc/mandriva-release, /etc/meego-release, /etc/angstrom-version, /etc/mageia-release all possibly being present.

Lot's of historical crud to clean up in *nix land, this is one of them.

The FAQ of that article is quintessential NIH, which I think summarizes the systemd project well.

Not just NIH; the condescension and strawmanning are well demonstrated as well. But yes, a wonderful example of the project's attitudes.

Can you explain why you think it's NIH? As someone who's written a lot of code to parse lsb_release (including an installer script that starts with "apt-get install lsb-release", because you can't even depend on lsb-release being installed on Debian/Ubuntu), I certainly agree with just about all of the FAQ in substance, if not in tone.

Both LSB and whatever Fedora was doing before were one small patch away from working for their purposes... so they created an entirely new format instead of trying to cooperate with anybody else.

What would the small patch look like?

As a consumer, the requirements I have are 1. always available, 2. machine-parseable without running a script, and 3. cross-distro (like lsb_release already is).

I think any patch that gets you that causes you to end up with /etc/os-release, although possibly under a different name, no?

It’s called “Pacing and leading”.

Why should the open source world throw out one of its greatest strengths: allowing mere individuals to make updates to the whole stack without the world burning.

Systemd's approach here is the kind of thinking that takes away 'on error resume next'.

Reply on news.ycombinator.com