Hacker News Reader

Comment by nailer | Hacker News Reader

I really like systemd. My blog has a 10 line .service file. My app has a 10 line .service file. My load balancer has a 10 line service file. They're way less brittle than bash scripts would have been. Stdout automatically becomes logs with the service name attached. It's nice.

View on HN

OpenBSD:

   #!/bin/sh
   daemon="/usr/local/sbin/daemonx"
   . /etc/rc.d/rc.subr
   rc_cmd $1

Though, it's Korn shell, not Bash, I suppose.

Reliable per-service logging was largely perfected by multilog.

Judging complexity by observing external configuration is nonetheless exceptionally naive. One must factor in how the key-value pairs are built into an internal representation and dispatched, which is where things become an issue. Non-deterministic boot order (man 7 bootup) is one of many undesirable trade-offs.

> Non-deterministic boot order (man 7 bootup) is one of many undesirable trade-offs.

Yeah, also something I found annoying with Upstart's induction into Ubuntu.

A predecessor at a place I worked at with a huge amount of traffic installed Ubuntu 9 on all of our memcache servers, and we found on a handful of other machines installed with Ubuntu 9 that non-deterministic boot order would randomly cause them to fail booting, and it was a coin-flip whether that would be a one-off problem or a forever problem requiring a reinstall.

When you have 4 8GB machines (this was some years ago) powering memcache for a billion requests a day you don't really want a power cycle to cause 25% cache misses for hours as you wrestle with hardware and boot config. :/

Upstart doesn't have a formal dependency system, instead it orders things based on named preconditions and postconditions called events. Though these can be used to fulfill ordering requirements, I'd wager the information wasn't granular enough for parallelization to be efficient.

Actually the main problem is that a lot of modern systems do not make their dependency graphs well observable, but instead treat them as in-memory ephemera that are constantly recalculated and cannot e.g. be written to persistent storage as known-good checkpoints.

This is one area where s6-rc gets it right. You have to compile a manifest of your service dependencies and consistently boot from that well-known state, so for any changes it must be regenerated.

serel was an earlier example of a service dependency system and parallel launcher which directly had you specify the nodes and edges of a directed graph as an RDF manifest. Clever.

For those who like to count lines, that's typically a whopping 9 line script, because it usually looks like:

    #!/bin/sh
    #
    # $OpenBSD: xdm,v 1.1 2011/07/07 18:42:17 robert Exp $

    daemon="/usr/X11R6/bin/xdm"

    . /etc/rc.d/rc.subr

    rc_cmd $1

for completeness' sake, here is the longest startup script on my OpenBSD system:

  #!/bin/sh
  #
  # $OpenBSD: nsd,v 1.9 2014/12/29 11:17:05 ajacoutot Exp $
  
  daemon="/usr/sbin/nsd-control"
  daemon_flags="-c /var/nsd/etc/nsd.conf"
  
  . /etc/rc.d/rc.subr
  
  rc_usercheck=NO
  
  rc_pre() {
          if ! [[ -f /var/nsd/etc/nsd_server.key ||
              -f /var/nsd/etc/nsd_server.pem ||
              -f /var/nsd/etc/nsd_control.key ||
              -f /var/nsd/etc/nsd_control.pem ]]; then
                  /usr/sbin/nsd-control-setup >/dev/null 2>&1
          fi
  }
  
  rc_start() {
          ${rcexec} "${daemon} ${daemon_flags} start"
  }
  
  rc_check() {
          ${daemon} ${daemon_flags} status || return 1
  }
  
  rc_reload() {
          ${daemon} ${daemon_flags} reconfig && ${daemon} ${daemon_flags} reload
  }
  
  rc_stop() {
          ${daemon} ${daemon_flags} stop
  }
  
  rc_cmd $1

37 lines of code (postgresql's weighs in at 33)

Your system now has millions of more lines of code that few people understand which are vulnerable to privilege escalation, unfortunately. :/

I mean, you could just use Heroku, for free, most likely, to host what you are doing. Should all UNIX become Heroku? What happens if your blog starts doing a billion page views a day? When one of the systemd processes is taking up 95% CPU, what do you do?

I'm not saying it's bad, I'm saying these are the type of concerns its' critics have. I am a fan of a lot of the ideas. I have downgraded several Linux systems which I needed To Work(tm) because of systemd being completely half-assed, and there are some great audits of its' code which make it appear to have been written by kindergarten students.

I'd take systemd over sysv init, atd, crond, three or four watcher daemons, syslog, various other semi-maintained tools that replicate each other's functionality, and around 30x 50-line shell scripts.

There we go with the false dichotomy that just won't die again.

It's pretty obvious you're not actually acquainted with the problem domain. Keep using whatever makes you feel comfortable, though.

Ehr, that's exactly how it used to be. Various sysvinit scripts that all differed across distributions. It's pretty appalling the way you respond though, just dismissing instead of trying to educate. I'm reading along and I have no clue what you're on about. Maybe you mean that there was other stuff out there? I care about what is on a distribution. E.g. RHEL had upstart, but in practice it was lots of large sysvinit scripts.

Serious questions:

Have you taken the time to examine how a Linux system that uses a replacement for sysvrc written in the last ten years actually handles startup? Or what services people who aren't running a RedHat distro actually run on their Linux servers?

Or have you simply either taken the systemd folks at their word about the "sorry state" of sysvrc replacements, or -perhaps- glanced at a >10 line startup script and recoiled in horror?

1. Yes.

2. Yes. Folks who aren't running 'RedHat' (sic) use systemd, sysvinit, or pretend to use upstart but as a thin wrapper to sysvinit.

3. No, I don't generally listen to Linux advocates as a way of evaluating something.

> 1. Yes. 2. Yes. ... 3. No...

Good. Far too many people in these conversations have poorly-informed opinions.

> Folks who aren't running 'RedHat' (sic) use systemd, sysvinit, or ... upstart [sic] ...

These are init and RC systems, not services. :)

> ...or pretend to use upstart but as a thin wrapper to sysvinit.

I dunno. The Upstart project page says:

"Upstart is an event-based replacement for the /sbin/init daemon which handles starting of tasks and services during boot, stopping them during shutdown and supervising them while the system is running.

It was originally developed for the Ubuntu distribution, but is intended to be suitable for deployment in all Linux distributions as a replacement for the venerable System-V init."

If I look at the file list for the upstart package, [1] I see that upstart provides its own /sbin/init, /sbin/halt, and -it seems- just about everything that's provided by the sys-apps/sysvinit package on my Gentoo system.

Also, sysvinit and sysvrc aren't the same thing. OpenRC -the default Gentoo RC system- is a sysvrc replacement. [2] It relies on sysvinit (or similar) to get its work done, and is an obvious improvement over sysvrc.

If you were aware of the distinction, and have the time, would you be so kind as to mention the top N worst things about sysvinit? If you weren't aware, and have the time, would you care to learn the distinction, and then bounce on back and answer the query?

Thanks in advance. :)

[0] http://upstart.ubuntu.com/

[1] https://packages.debian.org/jessie/amd64/upstart/filelist

[2] If I'm reading this right, Debian even maintains a sysvrc package: https://packages.debian.org/jessie/sysv-rc

Yep, I assumed you meant service managers not services. People not running Red Hat use the same services as Red Hat people do. Everyone runs web servers, mail servers, NFS, whatever else. If you actually meant services, then what's your point?

Re further requests for info, I don't really want to continue the conversation, because you should have apologised by this point and haven't. Don't presume to know things about other people, it's rude and looks poor when you're wrong.

> Don't presume to know things about other people...

That would be why I started my initial comment with questions, rather than presuming that I knew your background and knowledge and launching into poorly-formed and ill-founded accusation and/or assertion. :)

I'm trying to have a serious, level-headed discussion about a topic that is -inexplicably- sensitive for a few groups of people. In order to do so, I need to discover the pertinent information about the background and knowledge of my conversation partners.

> If you actually meant services, then what's your point?

I did. That would be why I asked about services rather than RC systems. :) I know that many people don't work this way, but if you can try to keep in mind [0] that I try hard to say what I mean, and leave nothing but white noise in between the lines of my statements.

Anyway, my point is this. You said:

> I'd take systemd over sysv init, atd, crond, three or four watcher daemons, syslog, various other semi-maintained tools that replicate each other's functionality...

which I took to be a little strange. It seems like this is your understanding of the typical set of services a Linux system runs, but it doesn't jive with how either I or my handful of professional sysadmin friends run our systems. We generally don't run atd, leave any FS watching up to the software package that actually needs it, avoid using tools that duplicate each other's functionality like the plague, and tend to not use unmaintained software... unless it's clear that that software is "done".

I've known several friends and acquaintances who have gone off to work for Red Hat. They've -to a man- come back with some really strange, very hard-line positions about how Linux systems are "supposed" to work. One of them vehemently insisted that -because of the complexity of dependency management and testing- a highly-flexible, rolling-release distro like Gentoo Linux was impossible. [1] Because of my personal experience with how Red Hat seems to have enweirdened some of my friends and acquaintances, and because I knew next to nothing about your background, I wanted to ensure that you had indeed stepped out of the Red Hat bubble and seriously looked at what other companies and distros consider a typical set of services and the features provided by and operation of their sysvrc replacements.

[0] I won't fault you if you can't. I have quite a difficult time remembering what I had for breakfast, let alone the rhetorical styles of the couple-dozen one-off Internet conversation partners that I've had over the past several months. :)

[1] Note that his position was not "Only a fool would attempt to provide guaranteed enterprise support for any and all arbitrary Gentoo Linux configurations because of $COMPLEXITY."! I can't argue with that position. I asked him if perhaps he misspoke and that maybe this or something similar was closer to his actual position. He vehemently denied that it was, and doubled-down on his original statement.

'Did you do (basic research)' is an impolite way to suggest someone did not. You have a lot to learn about communicating with others.

> 'Did you do (basic research)' is an impolite way to suggest someone did not.

I ask basic questions because I often have conversation partners who have not done their basic research. In the past, I've wasted a lot of time having mid-to-advanced level discussions about topics with people who -as it turns out- didn't have even the most rudimentary grasp of the fundamentals of the topic. I ask brief clarifying questions of this sort out of respect for the busy schedules of myself and my conversation partners.

More than that, I've found that presuming to know something about other people often leads to hurt feelings and wildly incorrect assumptions. :)

From the start, I've been behaving in exactly the way you claim that you want me to behave. I even explained -at length- the reasoning behind my first and second comments and my conversational MO.

After your reply to my first comment, you've failed to meaningfully engage in any further discussion, citing my failure to behave in the way that you want me to behave as the reason for your non-participation.

We're getting really far afield here, but do you make it a policy to apologise to people (who aren't paying you) who continue to fail to understand something long after you've made that thing clear to everyone else in the room?

Let's pull the last one from the list, because not only is it the most important, but it's the one which best demonstrates the attitude the systemd maintainers have towards interoperability, standards, and security. Can systemd's logger send to redis stores for aggregation? Can it email if there's an issue? Can I pull windows events into the same aggregation systems? Is there an RFC for syslog's file and network format?

These are all flaws in systemd with just one of those daemons you mentioned. Were I to get into init, I'd ask you about BSD-style rc files, which are tiny and very readable, as well as being shell scripts, so you can run more pre-execution tests to ensure your job will run correctly. With BSD atd and crond, I'd discuss how atd functionality has been merged in as a cron job. There are a lot of other solutions to sysv init than systemd. Unfortunately, the Linux monoculture sees bash as the only shell, sysv init as the only pre-systemd init, and the poorly configured rsyslogd distros ship with as the only syslog.

> Unfortunately, the Linux monoculture sees bash as the only shell, sysv init as the only pre-systemd init, and the poorly configured rsyslogd distros ship with as the only syslog.

People like to make fun of Gentoo, but it does many things right. ;) OpenRC and netifrc are sane and flexible. syslog-ng is the "default" syslog and comes with a reasonable default config file. Use of logrotate is strongly recommended, etc, etc, etc.

> ... BSD-style rc files, which are tiny and very readable ...

As someone who has an on-going project to replace 157 rc.d scripts from FreeBSD 9/10, I can report that the generalizations about the Mewburn rc.d system that are often made, in order to contrast it with the egregious mess that one can find in the System V rc clone on Linux operating systems, are not in fact strictly accurate.

Yes, most of the rc.d scripts are small. Yes, the Mewburn rc.d system did almost 16 years ago what people are erroneously characterizing as novel to systemd: namely taking all of the common management code out of the individual service definitions, leaving just a few parameterizations in some cases. Yes, there is a definite common coding style that is a breath of fresh air when coming from the Linux world.

But no, the scripts are not universally tiny. One problematic example currently of interest is /etc/rc.d/bluetooth, weighing in at over 360 lines. Others include the subsystems that are split across multiple scripts, such as /etc/rc.d/atm1, /etc/rc.d/atm2, and /etc/rc.d/atm3.

Nor are they universally consistent, /etc/rc.d/serial being an example of one with quite different configuration semantics to the key=value-in-rc.conf{,.local} norm.

And whilst one might rail against the problems of having a common script for SuSE Linux, Debian Linux, and RedHat Linux, all of which have different sets of init.d script helper tools (resulting in "portable" init.d scripts ending up being a case "$distribution" in...esac statement for pretty much every step), one can equally point to the fact that OpenBSD's rc.d system (https://news.ycombinator.com/item?id=8506460) has entirely different function and parameter names to the NetBSD one (as adopted by FreeBSD/PC-BSD and DragonFly BSD) as well as different rules about rc.conf and different variable naming conventions.

Be very careful about attempting to claim some kind of moral high ground on interoperability and standards. (-:

Also be very careful about notions that the Linux world sees the Bourne Again shell as the only shell, some rather silly statements by some people who should know better notwithstanding. The adoption of the (Debian-modified) Almquist shell as /bin/sh by Debian and Ubuntu, and its significant effect on bootstrap speed, is quite famous. And my experience is that the Z Shell is quite popular, for starters.

I don't bite. You're affirming many of the claims while listing a few inconsistent counterexamples. This does not invalidate the original claims though, as, relative to the virtually complete lack of standardization in writing SysV rc scripts (LSB notwithstanding), it is in significantly stronger shape. There are indeed disparities in rc.conf and rc.subr between the BSDs, though probably not any more or even less so than disparities in LSB initscript frameworks.

Really though, the primary reason to cite rc.d is to debunk the pants-on-head claim of shell script-based rc being intrinsically spaghetti and voodoo.

It's not a few. It's somewhere between a quarter and a third of the corpus. "few" is merely the level of outright extreme cases, like "/etc/rc.d/msgs stop" not doing what one might think at all.

I love the systemd principles, as they were outlined by Poettering in the initial announcement. Configuration files instead of boilerplate shell script (and worse, shell script with special comments), socket-based startup, massive parallelism, cgroups, the end of PID files...

I just wish systemd itself had stayed there and that the additional functionalities it keeps developing lived in separate projects with well-defined interfaces.

I find it disputable that parallelism is all that desirable. In systemd's case it requires the introduction of the dependency system with its multifaceted job queueing modes and transaction management that easily make up its most brittle edges, at least w.r.t. to its service manager, specifically. Seemingly innocuous configurations can result in cyclical transactions and dependency loops, and the internal graph engine bookkeeping these relationships provides very little debugging context at all.

Considering parallelism is about startup optimization, a much better option is to load a process from a checkpoint image, e.g. one taken by CRIU or DMTCP.

> ...a much better option is to load a process from a checkpoint image, e.g. one taken by CRIU or DMTCP.

Almost completely unrelated: Two things that came out of the LKML examination of kdbus during the 4.1 merge request were:

1) kdbus makes assumptions that break CRIU and similar projects

2) The kdbus people didn't seem to care much about this fact

So... the "ultimate" systemd setup may very well preclude the use of process checkpointers to speed up startup. :(

Wow, I wasn't aware of that. Do you have a reference?

Sadly, no. I actually might have mis-remembered that detail and made a liar of myself. :(

I tried to go back through the 4.1-rc1 kdbus merge request saga, but Gmane is so slow as to be unusable today and the only other lkml archive I can find that shows me the whole discussion [0] doesn't appear to do threads. [1]

A search for "CRIU" reveals Lutomirski mentioning that a particular bit of info passed around by kdbus ("starttime") would break CRIU. Greg K-H says "'starttime' has been removed for a while now. You sure that you're looking at latest?" Lutomirski says something to the effect of: "No, I haven't taken a close look at latest, just checked to see if that thing that I thought was really bad was still in it, and it was."

Later, I see Lutomirski wondering if kdbus would interact with '/proc/<pid>/map_files' in such a way as to break CRIU, but Michal Hocko says something to the effect of "Eh, it looks like it shouldn't, but I can't know for sure.".

Because I don't have the thread structure, I can't be sure that there weren't replies to any of those messages from folks who felt that there was a problem for CRIU. However, as it stands now, it's pretty safe to say that I fabricated a bucket of mud and threw it around.

Sorry. Mea culpa. :(

[0] http://www.gossamer-threads.com/lists/engine?do=post_view_pr... (Print dialog warning)

[1] I've just gotta ask: am I an idiot, or is Google Groups's mailing list search feature absolutely terrible? I ask it to give me all instances of "CRIU" in a particular LKML thread. It gives me a pointer to the start of that thread, leaving me to go through messages ~20 at a time to find the messages I care about.

Can you post your 10 lines for us to see?

Yes, please - genuinely interested.

Sure.

    [Unit]
    Description=Blah app

    [Service]
    ExecStart=/var/www/blah/bin/www
    Restart=always
    User=nobody
    Group=nobody
    Environment=PATH=/usr/bin:/usr/local/bin
    Environment=NODE_ENV=production
    WorkingDirectory=/var/www/blah

    [Install]
    WantedBy=multi-user.target

Starting as nobody is probably not a good idea if you have more than one service which does that. They'll be able to interact.

Could you please explain in more details the issues here and the right way to do it?

With two applications running under the same user, if one is compromised, it has ready access to the other.

Generally it's fairly easy to make a system user for an application if no special accesses are required - just make a custom system user for the application, and ensure it can access the various things it needs to touch (eg: logging).

... and if you want to learn more than has been already said, start with the famous design of qmail as a collection of mutually untrusting daemons that run under individual unprivileged user accounts, precisely to insulate each from the others using the normal operating system mechanisms that insulate one user's processes, files, directories, and so forth from another's.

A good question to ask yourself: "Presume that my daemon is compromised by an attacker ...

* ... what processes can it trace, debug, read/write the memory spaces and environments of, or send signals to?"

* ... what files, devices, and directories can it read and write?"

* ... what files, devices, and directories does it own, and can thus grant access to?"

Then ask: "How does the answer change if instead of all of my daemons running as the same (even if unprivileged) user they each run as different users, some (where appropriate) having no write access to anything and owning no files nor directories at all?"

Beautiful!

It's not even the number of lines that matters -- because start-stop-daemon could probably accomplish all of that in "1" gigantic line.

It's the fact that it's incredibly readable. Someone with only a passing knowledge of UNIX and no knowledge of systemd could understand almost everything in this file at a glance.

I can't say I agree. Does Restart=always mean the service is restarted if it is killed? Or something else? Why is there a block named [Install]? Why would my service file need to be installed? What is multi-user.wants? How do I know that this sorta-kinda-but-not-really corresponds to Runlevel 5?

However, the reverse view raises similar questions. How do you know what runlevel 5 is? (Hint: Someone with "a passing knowledge of Unix" knows that there's no coherent definition of run level 5. It's "power the system off" on some Unices, for example. Someone without a passing knowledge of Unix is probably at a complete loss as to what "level 5" could be.)

In actual fact, the multi-user boot versus single-user boot notion goes back to pretty much the beginning (https://news.ycombinator.com/item?id=10207674), and the name "multi-user" is not exactly difficult to cope with if one has a Unix background. It's not novel or a systemd-ism. It's been a Solaris SMF milestone name for over a decade. It was a concept used in the 4.1 Berkeley Distribution manual for init(8) in 1981.

> ...the name "multi-user" is not exactly difficult to cope with if one has a Unix background. It's not novel or a systemd-ism.

Sure, but I had to do a bit of reading to understand that

  [Install]
  WantedBy=multi-user.target

meant: "This service should start in the runlevel called 'multi-user'".

It also seems... backwards to put runlevel membership in a service definition file. Runlevels are an administratively controlled thing. Does systemd have two or more places a given service's runlevel membership can be specified, or is the only way to change a service's runlevel in its service definition file?

Yes, I know that the notion of "runlevels" is probably "legacy". However, assume that -on my OpenRC system- I have two particular runlevels, one for regular multi-user operation, and one for operation when I connect to hostile networks. Transitioning to this second runlevel shuts down all services that listen on anything other than localhost. Transitioning back brings those services back up.

Although this situation is a little contrived, this sort of flexibility is valuable.

The WantedBy value in the system definition file isn't an absolute (nor are the similar values Wants/Requres/RequiredBy/etc). Instead they are defaults.

What systemd actually does at startup, or when starting services individually, is look for symlinks. If I have a foo.service and a bar.service, and a symlink called foo.service.d/bar.service -> ../bar.service, then systemd will start bar.service automatically whenever I ask it to start foo.service. If it also sees a multi-user.wants.d/foo.service -> ../foo.service, then it will start foo (and also bar) whenever it enters the multi-user "runlevel". (This is almost exactly how sysv runlevels work, except that they're named rather than numbered, and they can depend on each other.)

I can make that symlink manually, if I want, or I can call "systemctl enable foo.service" in which case systemd will read the dependency information from the .service files and make the appropriate symlinks.

If I don't like how the distro or packager has set up the service file, I can override it. The service files live in /etc/systemd/system by default, but systemd also looks in /var/systemd/system for overrides. This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.

Putting the overrides in a separate directory is a pretty nice decision; it's always obvious what changes have been made, and upgrades never overwrite them. I can even put those overrides into version control and track changes to them over time.

> The WantedBy value in the system definition file isn't an absolute (nor are the similar values Wants/Requr[i]es/RequiredBy/etc).

Is this true of the identically-named keys in the Unit section?

> If I don't like how the distro or packager has set up the service file, I can override it. ... This is the real reason why systemd uses key=value stores for units; merging them is trivial while merging shell scripts is decidedly not.

You should look at how OpenRC handles service dependency specification. It's at least as flexible as the system you describe in systemd. :)

> Putting the overrides in a separate directory is a pretty nice decision...

Yep. At dependency graph generation time, OpenRC examines its master config file at /etc/rc.conf as well as config files in /etc/conf.d/ for -among other things- additions to the need, use, before, &etc declarations in the depends section of a given service's init script. To augment -say- the "use" list in the openvpn service, either add the line

  rc_use="my_custom_service"

to /etc/conf.d/openvpn , or add the line

  openvpn_rc_use="my_custom_service"

to the OpenRC master config file at /etc/rc.conf . If I -say- wanted to override OpenVPN's default dependency on the "net" virtual service, I would add

  rc_need="!net"

to /etc/conf.d/openvpn (or perform a similar transformation as before and store in /etc/rc.conf).

> I can make that symlink manually, if I want, or I can call...

To add the openvpn service to the "custom_init" and "default" runlevels, I would simply do

  rc-update add openvpn custom_init default

If I omit the runlevel, the service is added to the current runlevel. I could screw around with symlinks if I cared to, but why would I? :)

The OpenRC configuration processing system is substantially more powerful than I describe, handles the processing of OpenRC-standard config files [0] automatically, and lets you shift the complexity of a configurable init script that should be in a config file into a config file.

Because my overrides and augmentations to the distro-supplied init scripts are in text files stored in standard places, I -too- can easily put them in version control and track their changes over time.

[0] Which -shockingly enough- look exactly like what any Linux programmer would expect a non-XML non-JSON text config file to look like. ;)

Assuming you want all configuration in the init script, [0] here's the OpenRC equivalent. The USER/ENV/WKDIR variables are there for legibility.

  #!/sbin/runscript
  
  USER="--user nobody --group nobody"
  ENV="--env PATH=/usr/bin:/usr/local/bin --env NODE_ENV=production"
  WKDIR="--chdir /var/www/blah"
  
  description="Blah app"
  command="/var/www/blah/bin/www"
  start_stop_daemon_args="$USER $ENV $WKDIR"

Assuming you had a "multi-user" runlevel on your system and the service was named "blah", you would do the following to add that service to it:

  rc-update add blah multi-user

But, the "default" runlevel is probably the equivalent to systemd's multi-user runlevel. So:

  rc-update add blah default

Describe works as expected:

  $ /etc/init.d/blah describe
   * Blah app

[0] Real init scripts make use of OpenRC's machinery for processing config files in /etc/conf.d

Looks very similar to supervisor.conf

I'm not the same author, but I run several Webapps in systemd units, with the below service for all of them:

Note the @ in the name, that means I reuse it as webapp@api.domain.name webapp@frontend.domain.name and so on.

----8<---- webapp@.service ---->8---- [Unit] Description=%i webapp

    [Service]
    User=%i
    Group=%i

    # Note that the first argument of the command line (i.e. the program to execute) 
    # may not include % specifiers.
    ExecStart=/srv/webapp %i
    WorkingDirectory=/srv/%i/

    PrivateTmp=true
    Type=simple
    Restart=always
    RestartSec=2s

    [Install]
    WantedBy=multi-user.target

---- 8< ----

/srv/webapp is a simple script that will do "exec $1/webapp" which is the actual executable.

> /srv/webapp is a simple script that will do "exec $1/webapp"

I'm not trying to start a fight.

Why do you call out to a script, rather than having systemd make the call on its own?

It's for the same reason that java has to be invoked as

   /usr/bin/env $JAVA_HOME/bin/java

thus raising questions such as http://unix.stackexchange.com/questions/229523/ .

systemd does not permit variable expansions or unit parameters in the first word of ExecStart.

Assuming that your explanation applies to Spidler's situation, why not do

  ExecStart=/bin/sh -c "%i/webapp"

rather than using a one-off wrapper? Does that wrapper ship with systemd?

No assumption is necessary.

Roger.

Why do the thing Spidler described in his comment, rather than the thing I described in my comment?

Except you cannot extend them in any way, unlike a real bash script (init.d)

Try having a dozen different types of config-test for different programs, there is no way to extend ala ExecConfigTest

If you really want to wrap your service startup in a bash script, I don't think systemd precludes that approach.

But, that's kinda the thing...

Folks gush about how concise systemd unit files are.

For services that require no complicated pre-start checks or configuration, equivalent service start scripts are more concise and just as legible in either OpenRC or -AIUI- Upstart. (Look at my conversion of a simple systemd unit file [0] to an OpenRC init script. [1])

What's more, for jobs that need complicated pre-start logic of one kind or another -AIUI- systemd provides no way to bundle that logic with your service description file. You have to put that logic in another script file which is then called by systemd.

EDIT: I can't speak to Upstart, but even if your service has dependencies or needs to be started before or after some other service or class of service, [2] specifying them is similarly concise and legible. Add

  depend() {
   use some_optional_service
   need some_requred_service
   before some_service_or_service_class
   after some_other_service_or_srv_class
  }

somewhere near the top of your script. For most services, this adds three non-whitespace lines. :)

[0] https://news.ycombinator.com/item?id=10355588

[1] https://news.ycombinator.com/item?id=10357010

[2] For example: The network start scripts provide the "net" service. You might have a service that -for some reason- needs to start before or after the "net" service, but doesn't actually depend on it to start. This is what "before" and "after" are used for.

Upstart is pretty similarly easy to use in my limited experience with it. Definitely a nice idea to have stdout just automatically go to a log file.

Basically DJB did this in the 90s with daemontools, which many of us now use a revamped fork of called 'runit'.

You still write shell scripts, but you also still have control.

Again, I'm not entirely opposed to the ideas behind systemd, but I am opposed to basically every fucking Linux server distribution breaking because of things people are doing for the desktop, which began with upstart.

There is also no love left to lose between me and sysV init scripts.

I love djb's stuff - he has his own concepts of everything, though, including the FHS, which makes his stuff hard to run on regular Linux distros. If there was a djb Linux I'd definitely install it.

systemd has its own FHS requirements, as well: http://www.freedesktop.org/software/systemd/man/file-hierarc...

The only difference is acceptance.

But Nixos, which doesn't follow the FHS, still uses systemd...

Sure, much in the same way you can use daemontools without strictly following its FHS requirements. Or more likely you'd be using one of its many derivatives, as vanilla daemontools is rather old by now.

Those are not unique systemd features. supervisor provides that, as well, without requiring to replace the init system underneath.

Reply on news.ycombinator.com