Systemd Inhibitor Locks Deep Dive | Hacker News Reader

Systemd Inhibitor Locks Deep Dive

trstringer.com

53 pointszdw26 comments

The latest Kubernetes release 1.21 which was announced a couple of days ago actually uses systemd inhibitor locks for the kubelets:

"The kubelet attempts to detect node system shutdown and terminates pods running on the node. Kubelet ensures that pods follow the normal pod termination process during the node shutdown. The Graceful node shutdown feature depends on systemd since it takes advantage of systemd inhibitor locks to delay the node shutdown with a given duration."[1]

[1] https://kubernetes.io/docs/concepts/architecture/nodes/#grac...

I appreciate any article that approaches systemd as an API server that consumes dbus. I wish this was more common because I think the design is fantastic and should be emulated all over Linux userspace. So many services are spinning up random http, grpc servers or littering the filesystem with Unix sockets when dbus is so dang powerful.

> So many services are spinning up random http, grpc servers or littering the filesystem with Unix sockets when dbus is so dang powerful.

Or to rephrase, so many services are not depending on dbus :)

You can't win.

Though if your service is destined for deployment on a systemd system then that's priced in.

A service that runs as pid 1 inside a container would be a different matter.

Seriously if you do not need dbus, don't bother learning it. You will only have it in specific situations and most people don't know it as well as grpc or REST.

Except my thing on the user bus cannot talk to my thing on a different user's bus or the thing on the system bus. That all just works with those other methods.

Yes it can, why wouldn't it? You can make a program that connects to more than one bus. Though that would probably be crossing a security boundary. The simple way to make multi-user services is to just deploy them on the system bus and be done with it.

I'm not sure I'm supposed to engage with throwaway accounts but here goes.

Why can't a program on the user bus send a message to a program on the system bus and vice versa? Because dbus messages cannot transit between buses. That is why.

The workaround of straddling both buses is something I recently had a good look for. To see if that was a reasonable thing to do. I couldn't find a single example of a program doing this. I think that's because it's a security bodge trying to evade the intentional isolation built into dbus. Although if I'm honest I think it's an OK workaround as you can restrict the system bus access.

The problem I keep genuinely running into is the existing programs I want to talk to (and don't control) only sit on the user bus, very very rarely the system bus because they are gui programs tied into the user session. So my program is then forced to run as the same user. And even that's not enough because they need to run under the same user session, not just the same user. This restriction is a deal-breaker for my system service-type programs and is totally avoidable when using other communication methods (which those existing programs also provide, presumably because they realise dbus is often totally unusable).

The simple way is to use a Unix socket and that's why people continue to "litter" the filesystem with them. You can even get a bit of auth with them.

I'm sorry about the throwaway, it's a long story.

You're crossing a security boundary trying to access other users' GUI programs. The session bus is for programs that are private to the user's session. Accessing that is mostly equivalent to logging in as that user, you're not supposed to take control of all their programs as another user.

I don't really understand your solution, the problem doesn't go away when you use a Unix socket. You're still crossing a security boundary and now you have to build in those access controls into your program. If you don't really care about the security aspect it would be just as easy to tell the user to grant you access to the dbus socket for their session bus, or run another dbus instance under a different user and control access to that socket. You can publish the same interface on multiple buses at once, and build in your own access controls too.

Yep! dbus has really really good auth support so that users can talk to the system bus without needing to be otherwise privileged. The nicest is implicit auth where the dbus can just ask the kernel for the clients uid and authorize based on that. Very easy to implement and manage.

There is a lot of system services other than those provided by systemd on Linux that provide a DBus API - Network Manager, firewalld, storaged, sssd, packagekit, even the Anaconda installer.

This can be at least partially due to DBus API being a pre-condition for making the given service configurable via the popular Cockpit web management interface.

It’s unfortunate that dbus is used instead of a kernel-level IPC mechanism; despite how difficult it is to work with directly, Mach IPC is one of the hidden gems of NEXTSTEP through the Darwin family of platforms.

There were multiple attempts at getting a D-Bus-style IPC into the Linux kernel, none of which have particularly succeeded. It seems like the userspace solution works totally fine, and it allowed D-Bus to actually ship to people and take off - especially because it was easier to figure out which IPC mechanism had the right model and iterate on features.

And that's not to mention that a rich IPC mechanism in the kernel is necessarily a more intricate attack surface. Android's Binder RPC is in the kernel and it's had quite a few CVEs over the years.

Actually it's kind of funny that Linux, the monolithic kernel, is the one where IPC is in userspace, and Mach, the microkernel, is the one where IPC is an unavoidable part of the kernel. (Especially because the overhead of unavoidable Mach RPC seriously hurt its market success chances, and Darwin made Mach RPC avoidable by bolting on a BSD syscall interface, making Darwin in some sense worse than monolithic.)

I was really disappointed kdbus didn't make it into the Kernel (apparently for good reasons), as this would've solved so many issues.

What I'm still trying to wrap my head around is Varlink's [1] role in all this since then.

Is it supposed to make DBus obsolete at some point?

What level of compatibility towards DBus does it offer?

What's the driving force behind it's development?

How widely was it adopted so far (I've seen at least a few usage references in systemd, such as for the user/group API [2])?

[1] https://varlink.org/ [2] https://systemd.io/USER_GROUP_API/

What is the advantage of dbus over Unix sockets?

Apples and oranges.

DBus actually uses unix sockets (though I believe it can make use of other transports as well). If you're going to use a raw unix socket, you have to design your own RPC mechanism over it. For some things, that may not be a big deal, and may be preferable to pulling in a dependency like DBus. But for other things, not having to do that work might be a win.

DBus also allows for fine-grained permissions that you can't get with unix sockets without designing and building that mechanism yourself.

Dbus is mostly just a message queue for Unix sockets. Think Kafka or SQS but only for local services.

What's the advantage of REST when you have TCP sockets?

What does "github.com/godbus/dbus/v5" mean in a golang import? I've only ever seen the standard github.com/user/repo. I'm not sure what the v5 means. There is no branch/tag called "v5" in that repo, but there is one called v5.0.

Later in the code there is a line like

  conn, err := dbus.SystemBus()

but how does golang it know it's referring to the dbus package if there's a v5 at the end? I thought it infers the name from the last part of the URL if it's not prefixed with an alias.

Sorry if this is a stupid question.

> I thought it infers the name from the last part of the URL if it's not prefixed with an alias.

The name and the last part of the url are unrelated.

They're only usually the same by convention. For example:

    import (
     "fmt"
     "github.com/dimuska139/go-email-normalizer"
    )
    
    func main() {
     fmt.Println(emailnormalizer.NewNormalizer().Normalize("x+1@gmail.com"))
    }

The reason that I use "go-email-normalizer" as "emailnormalizer" is entirely unrelated to the url it's at or the name of the repo it's in. The only reason you use "emailnormalizer" is because that's the package line in all the files in that package: https://github.com/dimuska139/go-email-normalizer/blob/0432b...

Importing a go package will add an arbitrary identifier to the file scope, and you can't know that identifier without reading the package statement of the package you imported.

Here you go:

https://blog.golang.org/using-go-modules

Your v5 question should be answered by "part 4":

https://blog.golang.org/v2-go-modules

V5 isn't actually a file path. It's a permanent part of the import path in go.mod that indicates the major version.

So how do you cancel an inhibitor lock held by a stuck process?

You just kill the process, which causes the file descriptor to close. Cancelling the inhibitor lock by other means wouldn't do much, there's no way to get out of that situation without forcibly shutting the process down or disrupting its state.

Reply on news.ycombinator.com