poettering

Yes, I have.

Most of our tools that adjust files work really well offline. systemd-tmpfiles, systemd-sysusers, systemctl and so on all support --root= and --image=.

I really don't know what the problem is suppoaed to be.

Lennart

The 64bit issue is certainly an issue, but very much overblown.

First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is mostly a non-issue.

That said, sd-varlink/sd-json (i.e. systemd's implementation of it), of course is 64bit signed and unsigned integer clean when it processes JSON. More-over it automatically handles if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string.

Would it be better if JSON would have been more precise on this, yes. Is it a big issue? No, not at all.

You are writing this as if JSON was a newly invented thing, and not a language that has become the lingua franca of the Internet when it comes to encoding structured data. Well understood, and universally handled, since 1997.

A 100 digit number cannot be encoded losslessly in D-Bus btw, nor in the far majority of IPC marshallings on this word.

Having done systems-level OS development since 25y or so I never felt the burning urge to send a 100 digit number over local IPC.

Not that 100 digit numbers aren't useful, even in IPC, but typically, that's a cryptography thing, and they generally use their own serializations anyway.

Not really. We use two text based formats for logging: BSD syslog, and systemd's structured logging (which is basically an env block, i.e. a key-value set, with some tweaks). Programs generate text logs, journald reads text logs hence. Programs that read from the journal get the text-based key/value stuff back, usually.

(Yes, we then store the structure log data on disk in a binary format. Lookup indexes are just nasty in text based formats).

Hence, not sure what the Journal has to do with Varlink, but any IPC that the journal does is text-based, and very nicely strace'able in fact, I do that all the time.

[Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]

Varlink is not a message bus. Hence you should be happy?

The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world.

What does matter though are roundtrips. In Varlink there are much fewer required for typical ops than there are in D-Bus. That's because D-Bus implies a broker (which doubles the number of roundtrips), but also because D-Bus forces you into a model of sending smaller "summary" messages when enumerating plus querying "details" for each listed objects, because it enforces transfer rate limits on everything (if you hit them, you are kicked off the bus), which means you have to refrain from streaming too large data.

Or in other words: marshalling is quite an irrelevant minor detail when it comes to performance, you must look at roundtrips instead and the context switches it effects, instead.

Using JSON for this has two major benefits: the whole world speaks JSON, and modern programming languages typically pretty natively. And it's directly readable in tools such as strace. With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you. And if you tell me that that doesn't matter, then you apparently live in an entirely different world than I do, because in mine debuggability does matter. A lot. Probably more than most other things.

Lennart

It would make things more secure in this specific backdooring case, since sshd only calls a single function of libsystemd (sd_notify) and that one would not trigger the dlopen of liblzma, hence the specific path chosen by the backdoor would not work (unless libselinux fucks it up fter all, see other comments)

Dlopen has drawbacks but also major benefits. We decided the benefits relatively clearly outweigh the drawbacks, but of course people may disagree.

I have proposed a mechanism before, that would expose the list of libs we potentially load via dlopen into an ELF section or ELF note. This could be consumed by things such as packagae managers (for auto-dep generation) and ldd. However there was no interest in getting this landed from anyone else, so I dropped it.

Note that there are various cases where people use dlopen not on hardcoded lib names, but dynamically configured ones, where this would not help. I.e. things like glibc nss or pam or anything else plugin based. But in particular pam kinda matters since that tends to be loaded into almost any kind of security relavant software, including sshd.

Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That's trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this.

But let me stress two other things:

Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam/pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.

With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.

Nothing changed. You are just a victim of FUD on the Internet, my friend. That's all.

Yes, a tpm2 enrollment takes up one slot, the recovery key another, a fido2 yet another, a pkcs11 key yet another and a password yet another in any combination/subset you like.

With systemd you can enroll any string you want as "PIN" for tpm. There are no restrictions. Can be long, can be alphanumeric, contain weird chars, up to you.

systemd has a similar logic, i.e. a recovery key concept, but we made sure you can type it in wherever a LUKS password would work too, even on systems where systemd is not available but LUKS ist. The recovery key is output in yubikey's modhex alphabet which means you can type it in on many keyboards even without setting a keymap first, and will work. We also output it as qr code, in case you want to scan it off. All on all it should be as robust as a recovery key could be.

To my knowledge Ubuntu does not use the TPM2 PCR logic systemd provides at all, but their own.

Still would love if desktop terminal emulators would implement the zmodem receiver side, so that you can ssh into some host of your choice and just type "sz" to copy arbitrary files of your choice onto your local system.

We actually use something like the above. But thats not sufficient since we cannot set up the PAM session fully if $HOME is not accessible because we can't acquire a password for it...

Uh oh. This is implemented via a PAM module too. But it does substantially more than pam_mount so not sure what you want. It's like claiming that UNIX 'find' is NIH because 'ls' already can show directory listings.

Well. How frequently do you update your user record in /etc/passwd? I mean, if you do this a million times per second then maybe your are doig something wrong, no? I update my user record maybe every year when I update my password, but if you do that once per ms then uh oh

Yes this is a problem. To address this systemd-homed is careful to validate the user record enclosed in the volume first (which includes checking its signature against the keyring of accepted record signers) and checks whether the provided user password can unlock it. Only after this validation and that the fs actually matches the record we will mount it. Thus as long as the employer trusts its employees enough things should be reasonably safe.

(To make this happen the user record is embedded into the LUKS2 header metadata so that we can use it before mounting the fs)

Well, this boils down to: in a modern operating system, is it good design that an unprivileged user who logs in once can consume arbitrary runtime resources uncontrolled, unbounded forever, even after logout just because they decided to mask SIGHUP? I think not, I think the system should default to behaviour where unprivileged processes are clearly lifecycle bound, and when the user's sessions end they end comprehensively. I mean, other OSes don't really allow this unprivileged either, for good reasons: the lifecycle of the unpriv user's processes should be controlled by privileged code, and clearly be defined by the act of logging in and logging out in its lifetime.

It's entirely OK if the admin then opts out specific users or even all users from this behaviour, i.e. if a privileged players decides to liberalize unbounded, unlifecycled resource consumption for unprivileged players. But a default where unprivileged code can just stick around uncontrolled and consume as much as it wants forever is just a strange choice security wise.

i.e. I think the fact that SIGHUP masking is unrestricted, i.e. is not subject to privilege checks is the problem really. Something is unpriv by default that should be priv by default. And that's pretty much what this option in systemd provides you with.

Well. What is a distro for then if not for picking the most highlevel of defaults suitable for them?

We send SIGHUP btw. The kernel's own sending of SIGHUP is bound to the TTY concept btw, which is specific to TTY logins only, not graphical ones.

That said the question is not so much about who sends what, but more about whether a secure system should allow user code to escape lifecycle management or whether logging out means logging out and giving up all resources.

I think my comment above explained that already.

Aleksa,

I am not trying to get you to use our APIs. You talking about the cgroups APIs again, if I am not mistaken? As I tried to explain again and again: if you want container runtimes to manage their own cgroups then just set Delegate=yes in the unit file of your manager, get your own cgroup subtree, and you can do below it whatever you want, you do not have to call into systemd ever. Not a single API call, no C call, no D-Bus call, nothing. You get your own kingdom if you set Delegate=yes, and systemd won't interfere with that. This is extensively documented.

I wished you'd actually listen to what I keep repeating to you. We tried to be really nice to container managers, knowing that they disklike systemd APIs, so we put a lot of work in making the delegation boundary clean, so that they can be entirely systemd agnostic beyond setting the Delegate=yes boolean in their unit file, but alas, we just keep hearing the same nonsense.

The LXC/LXD people btw did get this right: they manage their own cgroup subtree now, and systemd doesn't interfere, and they don't link to or do dbus calls into systemd either.

I kinda like how a good chunk of the community complains we are "too monolithic", and the other side of the community complains we have "too many binaries". We can't win with you guys, can we?

Lennart

You make plenty of assumptions there, in particular that there was no communication about the session killing thing. Turns however there was. We informed downstreams about our intention and the reasons in detail, and we documented this for everybody else in NEWS. We also made sure there was an easy compile-time option to pick the default for this option, and then left the rest for the downstreams to decide: whether to default to on or off to this, taking in the information we got from us and from the rest of the community. If you think they made the wrong decision, then complain to them really. But seriously, you really just assume we wouldn't talk to anyone, without actually having any idea what it communication is really taking place.

First of all, as mentioned above, we made this compile-time as well as runtime-configurable, so that downstream distros can choose whether they want to make this opt-in or opt-out. Hence blame your distros if you picked it in a way you didn't like.

Moreover, this doesn't affect cron at all. Cron creates its own PAM session for each job it runs which means those jobs are independent from any real login session (i.e. ssh, graphical, tty login), and thus also don't get cleaned up by them.

This affected stuff that is forked off a login session and then stays around as "orphan" if you so will, i.e. with all session resources released, except for these processes that try hard to avoid clean-up (usually by double forking + detaching explicitly from any TTY/ignoring SIGHUP).

You know, because we knew this would be controversial we made sure it was both a compile-time option and a runtime option. Yes the upstream default of both defaults to on, but that's just upstream. We made it very easy and supported for downstream distros to switch between opt-out and opt-in of this option for their users. We have encouraged distributions to leave it on, but we were fully aware that for compatibility reasons this is something downstreams likely wanted to turn off, and most compat-minded distros did, as we expected.

Now I am used to taking blame for apparently everything that every went wrong on Linux, but you might as well blame your downstream distros for this as much you want to blame us upstream about this, as it's up to them to pick the right compile-time options matching their userbase and requirements in compatibility, and if they didn't do that to your liking, then maybe you should complain to them first.

(And yes, I still consider it a weakness of UNIX that "logout" doesn't really mean "logout", but just "maybe, please, if you'd be so kind, i'd like to exit, but not quite". I mean, that's not how you build a secure system. We fixed that really, fully knowing it would depart from UNIX tradition, but that's why we made it both compile-time and runtime configurable)

(Also, nobody has to "incorporate" systemd's library to avoid the automatic clean-up. In fact, there's no library we provide that could do that. What was requested though is to either run things as child of systemd --user or just register a separate PAM session, neither of which requires any systemd-specific library.)

Lennart

> Why exactly does systemd need to mess with the networking services?

It doesn't. All this stuff is opt-in. If you don't want to use it, don't.

What's a bit unfair though is that you imply that the various technologies where equivalent in behaviour and would be different only in performance. That's really not the case. The eBPF stuff allows configuration of per-cgroup, i.e. per-service or per-container access control. And that's a massive advantage. Classic Netfilter doesn't allow that: you can't tell it to permit all traffic to bittorrentd.service, because service/container/cgroup information is not available to it. Yes, people have tried to make that work, and there was some support for it in the kernel, but only for egress, and that made it pretty useless ultimately.

You know, if iptables would allow us to configure tables per-service, we'd happily hook that up. Such a design is in fact where we started from originally. But the fact is that such a scheme isn't really compatible with Netfilter's design and thus didn't happen. And that that's the case is really not systemd's fault, but a technical decision of the kernel community.

I know it is fashionable to blame systemd for everything, but seriously, eBPF+cgroups and all that kind of stuff is the result of the kernel's development scheme, and systemd only follows that and exposes what they came up with. If you don't like where kernel development goes, please complain about and to the kernel community, but systemd is just a consumer of what they do, and I couldn't care less if per-service firewalling is ultimately implemented based on netfilter, on nftables or on ebpf/cgroup. As it turns out only the latter ended up being capable enough to make this possible. And I accept that, and then made use of this.

Do note that the port forward hookups in nspawn (i.e. --port=) are actually implemented using netfilter, not cgroups/ebpf. I hope that illustrates a bit that we use what works and is provided by the kernel community, and don't really follow an evil scheme here.

Lennart

> Really!? This is (one of the) reason(s) why some of us greybeards loathe systemd (and just when I was starting to become "okay" with it, too).

I am not sure why you imply that there was some evil scheme behind all this. In reality, things are a bit different: one group at Red Hat decided to put an end to TCP wrappers (which btw is something other distros already did a while back, we are late at the party), and another group then offered to make a more modern replacement available, that could cover some of the uses, in order not leave users in the cold. I am not sure why you think that's an evil thing to do? Isn't it nicer to drop something with a suggestion how to replace it then to drop it without replacement?

Also, let's not forget: this functionality is implemented in the kernel, systemd merely adds knobs to expose this kernel functionality to the user. Hence, no, this functionality didn't really "show up in systemd". It showed up in the kernel, and systemd just wraps it to make it usable.

I mean, this is a dilemma, isn't it? Whatever we do you are unhappy, right? ;-)

Lennart