Systemd Is Not Magic Security Dust
agwa.name
agwa.name
It adds an easy to use method to restrict all of the daemons and services that someone might be running. Further, because the unit files are often shipped with the daemon/service, it'll do this by default on every install. Not just to the ones which have been looked at by a sysadmin. So a standard RHEL server by default has more restrictions than before.
The examples are pretty unconvincing IMO. The article talks about an email server. Systemd will provide an easy method to not have one service read the other one. In the examples it talks about email server and a web server. But completely ignores that someone will not just target that specific service, you could just as well compromise the NTP client and then move from there to another service.
Obviously systemd is just one small layer and security is not a yes/no thing. But why then talk specifically about systemd? The article seems to suggest that it's better to not use these abilities. This goes against the usual defence in depth security IMO.
The claim is not that it's better not to use these capabilities (which in any case do not require systemd; there are any number of ways to achieve the same thing). The claim is simply that systemd does not offer a substantial enhancement to system security, and that perhaps the increased attack surface offered by systemd itself outweighs any security benefit it provides to the services running under it.
1. Things should be written in different (safer) languages.
2. That a compromise in a mail server is already bad so having systemd protect the mail server according to him does NOT help
At least 50% is spend on those bits. Those are poor arguments to make. Systemd adds additional capabilities to make the overall system safer. You can do both. He stated nothing about an increased attack surface.
> 2. That a compromise in a mail server is already bad so having systemd protect the mail server according to him does NOT help
Right, the point of which is, as I said: systemd does not offer a substantial enhancement to system security
> He stated nothing about an increased attack surface.
He made reference to "baggage" which could mean many things. But either way the argument runs the same: systemd may offer some enhancement to security, but not a substantial one, not one that can only be achieved by using systemd, and not one that's worth the cost of using systemd.
So does chroot. And so does jails. And posix capabilities. And cgroups...
It's not perfect, it doesn't do everything. But is your suggestion really not to make use of systemd abilities because maybe something else could be used? How does this not go against the defence in depth theory?
Try and use (filesystem) capabilities. Now apply that security fix: capability is gone. Further, that was just a change made to your machine(s).
It seems much easier to improve the security on all fronts instead of going for some idealistic super safe solution that either takes ages or doesn't happen.
Example: pretty nice to restrict that Jabber daemon written in Java from accessing too much of the filesystem.
There are similar arguments (pro/against) docker where 2 camps, one saying it's poison & bloat the other the best thing since tabbed browsing.
People get so riled up because they're unwilling to acknowledge that they're just tools that make lot sense for solving specific issues. Everyone keeps looking for silver bullets. Problems start when one camp tries to become the de-facto standard (not just the tool but also the processes that come with it). Also people are way to scared of code-forks IMHO. The "fail quick, fail fast" mentality should apply here too. Some healthy competition and a willingness to let certain projects just wither and die should be a good thing ...
There's also the fact that it's written in C. Writing security-sensitive software in C in 2016 is, in general, a grievous mistake. No, other languages are not perfect; yes, other languages have their own issues; regardless, C is worse than most. Buffer overflows, memory management, lack of type safety — Just Say No™.
Is it though? Although I can't think of a lot of vulnerabilities to be concerned of in PID 1, I can think of a few (perhaps something to do with reaping zombies?). Why even allow for that possibility? I see no reason why init(1) couldn't be written in Lisp, or Python (although the speed might not be great), or Go, or Rust, or Haskell, or ML.
If you want your init to integrate RC or process management (many do), sure, write it in a more complex language (but depend on external libraries minimally, and make sure it's statically linked!), but for the idea, minimal init, C is perfectly fine.
A run control system, which should be a completely separate project, could easily be written in a higher-level language than C. For instance, the traditional Linux RC system is written in sh, and the official GNU RC system[2] is written in scheme.
Daemon run control isn't a domain where performance is crucial; transparency is much more important there, so the argument for using a higher level language to control daemons is pretty strong.
[citation needed].
Gladly, good sir.
https://cfp.systemd.io/en/systemdconf_2016/public/events/21
Sit back, and witness the insanity.
> That presentation is from the Cockpit project, which isn’t part of systemd and doesn’t even share its main developers. It’s as much “remote access to systemd” as using SSH and then invoking systemctl.
and then further states in the 2nd blog post:
> I certainly don’t buy his original argument (which he hasn’t repeated in his second post) that tools like Cockpit should change the view of systemd. Just like SSH or Remote Desktop in Windows, Cockpit is the tool providing the remote, privileged access; systemd is just on the server listening locally. Blaming systemd for a Cockpit-based compromise is like blaming the “dnf” utility for removing the web server packages after someone gets root over SSH. (Cockpit is neither developed by the systemd team nor installed by default in any distributions.)
So..
> Sit back, and witness the insanity.
Who is acting insane here? A systemd developer, or people spreading FUD?
So yes, systemd isn't at fault, and I did miss it, and, to some degree, spread FUD, for which I am sorry. However, it's still a very silly idea.
@munin[0] did a pretty good illustration on why both pro/con systemd camps might be right about their reasons:
→ So, systemd is kind of a perfect microcosm of all the 'problematic' behavior in tech, all at the same time. It's a project that is dedicated to novelty and a specific set of goals - mostly speed-centric - above all else. And, to its credit, it -does- make certain linux systems boot faster than other, competing init systems. However. First, the way in which systemd has progressed has taken specific advantage of certain problems in the open-source community. Namely, that there are many projects that have little or no attention paid to them, despite being infrastructurally critical.
→ So there is little desire by various linux distributors to make the effort to maintain them - and when someone shows up offering to replace and maintain that functionality, taking the responsibility off the already overworked maintainers' plate? That's attractive.
→ Open source maintainers are always in a deep technical deficit trying to get these old bits and pieces maintained, so they're eager to get the help, and don't look too closely at what his 'help' entails.
→ And unfortunately, what this 'help' entails is the highly toxic systemd community - and I use 'community' in the loosest sense, because it really works out to being a sort of cult, spearheaded by a specific individual with the ultimate power of acceptance or rejection over anyone else's participation in the project.
→ Which is really unfortunate, because that guy's got an ego the size of Manhattan. And he continually refuses to take criticism over his design and implementation choices - take a look at what's left of any bug reports.
→ I say "what's left" because he has a history of purging anything he considers to be "un-useful" - that is, critical - from the archives
→ Now, certainly, you -could- go ahead and fork the project! Except now you have a giant codebase and no community to work with to fix it. You could always convince the distros to abandon it! ...except you're now up against a bunch of overworked people who, frankly, won't care. And, worse, now that it's a de-facto 'standard' in the linux world, you have a whole lot of institutional inertia to work against to try to replace it, and - unlike when they replaced init - a dedicated group of people who are utterly convinced they're doing the right things advocating against rolling back the changes. And, worse, there -are- some good points. SysVInit -is- grody as all get-out.
→ However, because of their dedication to novelty above all else, they're making not only all the same mistakes sysvinit had to learn...
→ [ And because of ego, rejecting these mistakes as being 'un-useful criticism' ]
→ But they're making whole new kinds of mistakes - things like https://cfp.systemd.io/en/systemdconf_2016/public/events/21 … which is just -staggeringly- WTF.
→ It's this whole fun trend of "fail fast" that, sure, looks great in a startup producing some new kind of app for making your phone go yawp
→ But it's not -really- a very good model for infrastructural type concerns - the things that need to be, by reason of their importance - conservative and slow to change. You want your infra to be -reliable-, not "full-featured" - at least if you've any sense.
→ So: You have the trend-chasing guys who show up to solve "all your problems" at the cost of making mistakes that could be seen a mile away. Eventually leaving you far worse off and making a huge mess that will be -extremely- expensive to fix when a reckoning comes. All in the name of some "change" that's needed from the status quo and without understanding why the status was quo. What's the solution here? Well, for one thing, take a look at the fine print when Mephistopheles shows up offering to take care of all your problems. It's too late for Linux - most Linuces are pretty much doomed at this point into becoming utter travesties that make WinMe look reasonable. Though the Devuan and Alpine folks seem to have some good impetus behind them.
→ Consider carefully what consequences are going to show up from adopting these new and nifty 'features'.
→ Consider that there is a very large difference in requirements between core infrastructure and user-facing things.
→ [ Because user-facing things can fail fast and be updated fast, but core infra is much more expensive and time-consuming to do either ]
[0] Source: https://twitter.com/munin/status/781257878321582080
Sorry about the above format, was too lazy to do a 'storify' or 'tweetlonger' also hoping he puts that into a blog.
On my laptop, it's fine, I guess, though even there I aesthetically dislike parallel service initiation; I'm glad Jessie still lets you replace it with a real init system, though I wish epoch [1] were packaged.
When is this a problem?
> systemctl start foo.service && systemctl status foo.service
If you need to do that, the problem is with foo.service, not systemd. If foo.service fails to start, the right way, then "systemd start foo" will fail. This is true of any init system: if a service starts but then bails out after success has been reported to the user, it's too late to change that.
> static /dev tree
What's insanity-causing about a dynamic /dev tree "on a server"?
You're joking, right?
Sysadmins need a deterministic boot order, period. If I want services to activate on a request I'll use inetd, but I haven't wanted that in years.
> If you need to do that, the problem is with foo.service, not systemd.
This isn't about assigning blame, this is about my server being usable. If a service fails to start I need the command that called it to fail also.
> What's insanity-causing about a dynamic /dev tree "on a server"?
Again, it's non-deterministic. Worse yet, it's declaratively configured in a million and one udev rules instead of one imperative run control script (which would at least be better, though still a bad idea) I need the devices on my servers to always have the same names and numbers, and obviously a static tree which doesn't expose any surface for my error or somebody else's attack is strictly better than a configurable system that does expose those surfaces.
My run control scripts start the services that need to be started, in the order they need to be started, one at a time, because at 3 am with the alarms going off, that is transparent to me or to whoever has replaced me after the tragic bus accident. Sysadmins get this, which is why so much of the pushback against systemd came from us. Distribution maintainers love systemd, and I get that, but I'm not a distribution maintainer. So now my site-local imperative run control scripts replace an upstream declarative config system rather than a distro-maintained set of run control scripts. C'est la vie.
If your services need to be started in a particular order to work your services are broken.
I've seen this sort of setup over and over again. Some custom 'startservices.pl' script that starts services "one at a time" in the "right order" like application server -> web server.
Then one day the DBA restarts the database that lives on another box, the app server crashes and the entire site is down.
So you get paged at 3am and run your startservices.pl script to fix the site. Great job, pat yourself on the back.
Meanwhile, a site run by admins that use process supervision had a 5 second outage until the app server process was restarted automatically.
You can tell systemd that one service depends on another, but it shouldn't really be needed. This doesn't even have anything to do with systemd. You can do the same thing using runit.
> Sysadmins get this, which is why so much of the pushback against systemd came from us.
..."us". No. You don't speak for everyone.
Umm... that's possibly the silliest thing I've read on HN, which is going a ways.
I don't know about you, but I like for my web server to come up after the NFS share it reads from is mounted. YMMV.
Why does this matter? Your load balancer or service discovery layer should detect that the web server is not functioning properly and take it out of rotation. What do you do when your NFS server has an outage?
Even if you DID care about that sort of thing, systemd has a RequiresMountsFor option:
RequiresMountsFor=
Takes a space-separated list of absolute paths. Automatically adds dependencies of type Requires= and After= for all mount units required to access the specified path.
#!/bin/sh
# web/run
if [ -d /web/root ] ; then
exit 2
fi
.. normal start commands here
I'm sure your hacked up scripts handle this scenario or anything else that can go wrong. Or maybe you get paged at 3am every few days when something breaks. YMMV.
Yes, I agree there are complicated ways that systemd and other rc systems can, largely, emulate the flexibility I get from writing a bash script to start the services I want started. I just don't really want them.
Yes, I used to work with people like you. "What's wrong with this method? we've been using these scripts for 10 years!"
And every time the database restarts the app server crashes and the site goes down. And for 10 years this has been seen as perfectly normal. After all, it's so simple! At 3am all someone has to do is login and run some site specific bash scripts that someone hacked together 10 years ago.
Nope. I've also never had these phantom poorly behaved forking daemons leaving orphan processes all over the process space that people claim drove them to systemd. Like I said, YMMV. What I've built works really, really well, and can be picked up tomorrow by anybody who knows sh.
Sounds like a decent description of LKML.
Here is a response to the OP response, by the way: https://medium.com/@davidtstrauss/ayer-vs-systemd-part-4-832...
I myself am only a hobbyists sysadmin but I -love- how easy it is to make daemons that actually start and initialize in the correct order (no more bash script loops waiting for a socket to be active!), have services run as their own user enforced at the init system level while still accessing named ports, while also having the added minor benefit of parallel startup. In my hobbyist use, I have not encountered any bugs/errors and I've been running it since debian first shipped it in testing.
Obviously I don't use it as much as real sys admins, but the point remains that despite its negatives it brings plenty of positives as well.
More to the point, I have now worked for two large companies which are holding off on upgrades to systemd-based distro versions in an effort to avoid having to deal with systemd.
It's such a complete paradigm shift, and so opaque, that not one of the sysadmins I've worked with is comfortable betting their jobs on it.
Stupid things like the infinite loop via systemd-notify do little to ease that concern.
And how long did it take you to figure out the same thing with sysv init when you were just starting out? I imagine it was more than 0 min. Plus now that you know where that config file is you don't have to look it up again so that cost was once...
With my anecdota, I have had no end of problems with bash scripts and sysv init including non deterministic behavior. Anytime i'd run into problems I'd end up having to write complicated bash scripts myself to fix them. Note all of this is to get my applications to run, not fixing linux distribution package inits (which are generally semi well tested). If you look at the distribution package inits, they are typically as complicated as the ones I had to write just already done for you. Its a mess.
The answer is, and always was, '/etc/init.d'. Upstart lists its directory ('/etc/init') in the first sentence of its second paragraph.
SystemD, on the other hand, requires that you first know that they are now called unit files with a .service extension. Once you have that, you can then attempt to find the man page which finally lists the possible directories they can be stored in as any one of of '/etc/systemd/system/', '/run/systemd/system', '/usr/lib/systemd/system', '~/config/systemd/user/', '/etc/systemd/user/', '/run/systemd/user/', '~/.local/share/systemd/user/', or '/usr/lib/systemd/user'.
Unless you're tracking down a port opaquely owned by systemd, in which case you then also need to know that systemd refers to them as 'socket activation' services, and that those files end with .socket instead of .service.
> I have had no end of problems with bash scripts and sysv init including non deterministic behavior
People are running into the same determinism problems with Systemd, so it doesn't appear this is a step forward.
* http://jdebp.eu./FGA/systemd-documentation-errata.html#Missi...
> in the absence of any overriding environment variables
Sigh.
> Sigh.
Almost as if they are deliberately trying to make it hard...
Characterising the mere existence of an external HTTP gateway to systemd (and other components) as a "staggeringly WTF" mistake made by the systemd team is at best unreasonable. That's like saying sysvinit sucks because Webmin exists.
As for myself, I use Arch. It's systemd by default, but people have put good effort into making switching easy, so I'm on openRC, which is great, and doesn't cause any problems. The piece of software that breaks most on my system is Steam. Which is to be expected.