Relative shell script includes with realpath on FreeBSD | Hacker News Reader

Relative shell script includes with realpath on FreeBSD

patmaddox.com

83 pointssigna1163 comments

As someone who has fought a long time against holes in POSIX sh (cf https://git.sr.ht/~q3cpma/scripts README with links to the Austin group Mantis), let me list my biggest pet peeves (including this one):

* mktemp -d (mktemp itself is available via POSIX m4; the noclobber solution mentioned is moot, since FIFOs break it https://austingroupbugs.net/view.php?id=1364), very much needed and available on GNU, *BSD and MacOS.

* local or typeset: I think only ksh88/93 doesn't have it without using the non-standard "function" keyword, a runtime check with potential alias creation is my only "solution" (https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a9534469089...)

* realpath/readlink -f/-e: as said by some, realpath is getting standardized in the next POSIX; can be emulated via a cumbersome function (https://git.sr.ht/~q3cpma/scripts/tree/b21de0bfb0a9534469089...) but since you either put that massive block in your script or need realpath to reliably source it from another file...

* All the "newline in paths" mess: I gave up, just too many blind spots.

Another major hole (in all shells!): there's no way to perform quote-aware splitting of a string in a restricted manner. This is especially egregious given the lack of array variable support.

`xargs` is close, but only handles a few kinds of quotes, not e.g. `$''` which is the best kind of quoting, and notably does not adapt to the current shell.

`printf %q` is also pretty important to avoid injection attacks, though if you don't care about ugliness or performance you can hack it together pretty easily (but beware of trailing newlines in subshells!).

(also, I am obligated to point out that BSDers are lying when they say they're using "Bourne shell". FreeBSD uses a fork of `ash`, related to `dash` and `busybox ash`, but all 3 have evolved independently. OpenBSD uses a derivative of `pdksh` (other BSDs also ship derivatives but not as /bin/sh). NetBSD has their own from-scratch shell, but NetBSD is dead and irrelevant. I have a whole tree of shells; `ash` is notable for being the most unpredictable within the tree.)

Indeed, even the script you linked is not good enough; it does not deal with trailing newlines in the file or directory name, and the ls parsing will fail with a file like 'my malicious -> file'.

Here's a good explanation from SO that I happen to have in my bookmarks:

https://stackoverflow.com/questions/29832037/how-to-get-scri...

Thanks, I will use that from now on as a reference when somebody asks me why I don't like to program longer things in shell.

He is still missing the double quotes, paths may contain spaces

Yep.

    LIB="$(dirname "$(realpath "$0")")"

It seems like overkill, but just get in the habit of quoting anything and everything.

You can technically omit the double-quotes during a scalar variable assignment (as long as it's not preceded by a modifier such as `export`), so the outermost quotes are not needed above.

https://unix.stackexchange.com/questions/68694/when-is-doubl...

The extra quotes don't hurt, but also: use shellcheck. It will tell you if you've omitted necessary quotes.

> The extra quotes don't hurt

I have the habit of quoting everything, that way I don't have to think about any corner case where I can skip quotes, and thus conversely never miss anything where I should quote.

I do the same with curly braces, that way I don't have to think whether "${foo_bar}" or "${foo}_bar" was meant, and makes later adding or reading string manipulation (e.g "${haystack/needle/replacement}") or arrays (e.g "${foo[42]}" or "${foo[@]}" that much more consistent: variable references are always bounded by ${} which makes code easier to scan.

I apply that to "${1}" and "${@}" too whenever possible, unfortunately bash 3.x doesn't quite like that so it's not always the case (I can't decide which one I hate more: macOS for the refusal to either update bash or outright drop it† and just have a plain posix sh + zsh, or GNU for the viral license move which is highly detrimental in practice for that case).

I also often quote "semantically", e.g:

    # instead of:
    PATH="/some/path/${foo}/bar:${PATH}"
    /some/command "--foo=${bar}"

    # I'd rather do:
    PATH="/some/path/${foo}/bar":"${PATH}"
    /some/command --foo="${bar}"

† I guess there's a crapton of bash scripts that have been written and possibly embedded in apps or installers, so backwards compatibility won here.

Are you me? :)

I actually personally use `declare` frequently, particularly to make variables readonly. So in my case the quotes are necessary. Of course this has its own issues, in that subprocess exit codes are hidden, so you often need to assign then declare.

But I quote everything, always, because then I don’t have to rely on remembering when it’s necessary and when it isn’t. Just do the safe thing every time and you don’t have to think.

no, you should only quote things that you want quoted. ("duh")

the caveat being if you actually do want things expanded then you should not quote them.

   for i in $(seq ...)

comes to mind...

> paths may contain spaces

what kind of savage puts spaces on their paths? Failing in such cases is a neat feature it would seem.

I never understood why common filesystems allow this foolishness. There are already other semantically meaningful characters in paths that are forbidden, like '/' or '\0'. Disallowing bare spaces in filenames is such a natural feature that it makes no sense to deal with all this stupid quoting pain. Any space that is entered by a user in a filename context should be transparently translated to a unicode non-breaking space. Most users wouldn't notice, and shell scripts authors would be much happier!

Because space is one of the most common characters in many languages, and people want to write stuff with spaces.

> Most users wouldn't notice, and shell scripts authors would be much happier!

They would, because people want to use spaces, and they wouldn't, because there's a bunch of characters you need to quote for (in POSIX sh/bash).

The solution is to realize it's not 1986 and evolve shell scripting beyond that. Fish and zsh have done that. Probably others as well. It's not even that hard and doesn't even need to woefully break compatibility.

Oh, and even if you did somehow convince the Unix world to outlaw spaces and other special shell characters by some super-human feat of charisma, you still haven't really solved anything because there's no way in hell you're going to convince the Windows or macOS people to do that, so you're still going to end up in trouble when your spouse or grandma or boss gives you their USB drive, or sends you files (which you can conveniently no longer store), or whatnot.

Might as well argue for the moon to be painted green. It's just not going to happen.

Failing in the way that it is virtually not fixable (e.g. Windows user names) or destructive (e.g. removing some unrelated directories) is not neat. It is even often said that Windows had a space in `C:\Program Files` (now `C:\ProgramData`) in order to make sure that app developers have to handle paths with spaces correctly. Don't know if it's indeed true, but at least it's believable.

It's much worse than this. On Unix '\000' (NUL) and '\057' (/) are the _only_ things that are not allowed. Anything else, whether it's valid in any encoding or not, is allowed.

In order to test my shell scripts, I have a directory hierarchy of filenames generated from /dev/urandom, which contains a sub-structure with filenames in unary, using only the newline character.

> what kind of savage puts spaces on their paths? Failing in such cases is a neat feature it would seem.

windows noobs (as in embedded development), but more likely grabbed music.

> I never understood why common filesystems allow this foolishness.

Because back in the days of K&R the kernel just blatted out whatever bytes it was given, and the C/Unix folks refuse to even consider that old APIs might have problems and need changing.

perhaps someone wants to put a script in their icloud directory on macOS: ~/Library/Mobile Documents/com~apple~CloudDocs

Τĥιs ñåmè/ įß ą váĺîδ POSIX paτĥ

Looks good to me. Assuming that the spaces are unicode nbsp.

your entire reasoning is just that your UI (a shell) cant work with spaces. literally unix brain damage. one UI supports this one supports that, neither should have any pressure on how data in the system is formatted. your tool is just broken.

Yeah that made me really anxious while reading this. That's the first thing I hammer into fresh hires and interns who are "lucky" enough to end up dealing with shell scripts.

In addition to other details noted about escaping/quoting and the portability of realpath, it's important to realize that those paths by default are not relative to the invocation directory, they're relative to the current working directory, which happens to be the current working directory of the parent process initially. Also, for a shell script it is relatively reliable to rely on $0, but not guaranteed by any means: for example, an obvious way to subvert this would be to pipe the content of the script into a shell, though there's surely plenty of ways you can think of that this could be broken.

One approach to this problem that’s been fruitful in Nix ecosystem is ~packaging scripts and baking in absolute paths at build time. (Generally by rewriting references in the script to absolute paths or generating wrappers to set PATH or BASE or DIR or whatnot.)

Most scripts I write cd into the directory they are located in for this reason. (and so that nohup.out files don't clobber each other in my home folder) Hopefully I can keep this in mind if I ever have to symlink them.

+1 - it really irritates me when some script insists that I change into its directory in order to work. Its just lazy script writing.

It can also be annoying when a script changes directory itself. If anything the script does is relative to the current directory (interpreting a relative path, creating output files), you now can't have that be relative to your current working directory.

Best practice, IMHO, is usually for scripts not to touch the current directory, and to be agnostic to it.

I didn't know about this. I'm writing bash scripts in the scripts-to-rule-them-all fashion for work and this can help DRY things up

Wouldn't

  readlink -f $0

work to get the actual location of the script, and from there get the base for relative scripts?

The -f flag is a GNU extension. Using realpath is more portable.

That's a portable sh solution, isn't it? Like, yes, it does work on FreeBSD, but it isn't exclusive

The portablest solution I have found is:

    rundir=$(cd -P -- "$(dirname -- "$0")" && printf '%s\n' "$(pwd -P)")

This isn't a portable solution at all because it doesn't solve the problem of the script being symlinked. While it does resolve directory symlilnks, that part is usually not even needed as finding a helper works just as well with a symlink to the directory containing the script as it does with the resolved directory. What doesn't work is a directory containing a symlink to the script which your "solution" fails to resolve.

Additionally (and the article doesn't solve this either), when invoked through $PATH, $0 will not be a full path but only the name of the command. In that case you want to start with "$(command -v "$0")" instead of just "$0".

No, $0 is a full path when invoked via $PATH.

I'm curious: why the `printf '%s\n' "$(pwd -P)"` ? wouldn't just `pwd -P` achieve the same?

Actually nothing. The format seems to have come from a similar line that builds the script path..

    printf '%s/%s\n' "$(pwd -P)" "$(basename "$0")"

It's just been in place, in so many places, so long it's become unquestionable. So yes the `pwd -P` in this case will suffice.

I've been using just `V="$(cd -- "$(dirname -- "$0")"; pwd)"` for ages, everything quoted, several OSs, never had an issue with the output that required me to format it again.

This approach breaks if you run the script from a symlink.

I think the closest portable solution we have would be basename instead of realpath, but I don't think that takes care of the symlink problem described in the article.

POSIX has no realpath(1), so no.

*Yet.

Will be added in the upcoming revision: https://www.austingroupbugs.net/view.php?id=1457

20 years ago it was a "BSD thing", but it seems to be more wide-spread these days; even my Alpine busybox system has it. I'm sure there are some platforms without it, but there are also platforms without some POSIX tools – it's never a guarantee.

All the main platforms have realpath(1). It was added to GNU Linux over 10 years ago now, to aid portability, and be a more natural command to access this functionality than readlink(1)

Saw a comment on another site mention this, but NetBSD won't get it until 10.0 (not yet released) and OpenBSD got it in 7.1 (released last year). Of course I'm of the mindset that building complex shell scripts is simply an exercise in masochism so this all seems like a solution in search of a problem.

Not really. macOS until version 12 doesn’t have realpath(1), it was only added this year. You can’t rely on it for portable code.

To clarify, it was added (late) last year, with macos 13 (Ventura).

FYI realpath was not available by default on MacOS until very recently as others point out: https://news.ycombinator.com/item?id=38514413

Prior to that you would need to do 'brew install coreutils' to get realpath.

I reject the symlink problem as not real, and i think the solution is harmful.

Here's my expectation: if a script is linked into some location, and invoked at that location, it should behave the same as if it had been copied to that location.

If it wouldn't work if it was copied, it isn't reasonable to expect that it should work if it's symlinked. There's no point catering to that use case.

Moreover, let's say i symlink the including script, then copy the included script to the right relative position, then edit the included script. I would expect that running the symlinked including script should include the copied and modified included script, not the original.

In general, i think programmers are far too eager to apply realpath or readlink -f to paths. If a path has symbolic elements in it, it's because someone has put them there on purpose. Don't just remove them. It's actually very rare to really need to know the canonical path to a file.

Where do you see the harm in the solution?

To your other point, I think this is a valid use-case and do not agree with your rejection. I have had experience with such very examples and submitted PRs to add `realpath` to fix them. One example, a linting app I installed put itself in `/usr/local/<linter-name>/bin/<linter-executable>`. It was really painful to type this full path every time (even with tab completion). I created a symlink from `/usr/bin/` to the executable, and voila, the command is on my path now and easy to use; except the script does not expect itself to be a symlink and broke. To work around this, I created an actual script in `/usr/bin` that could invoke the utility (which then worked because it did properly detect path at least with $0). I don't think that extra script should have been necessary. Further, if I only updated my path, then `dirname $0` fails (returns '.') and that was not even a solution.

Second question, as a counter-point, couldn't someone also reject the expectation that scripts should work when invoked from any path at all? That is saying, as a script writer, when you file a bug report, I can respond "closed, this script requires a `cd` to the install directory and only then can be run. RTFM!". At what point is this just ignoring UX?

The harm i see is what i describe under "moreover" - inconsistency with respect to copying.

In your linter example, i think the most normal solution would be to add /usr/local/<linter-name>/bin to your PATH. Your solution, of a script which just does exec /usr/local/<linter-name>/bin/<linter-name> "$@", also seems fine to me - quite simple, and less likely to cause weird behaviour than a symlink.

I'm not sure what point you're making at the end. We agree that a script not working when invoked from anywhere is bad UX, because people expect to be able to do that. I think that scripts not working through a symlink is not bad UX, because people can't expect to be able to do that, because in many cases that doesn't work!

This has bugged me for a long time too – on Linux you can use readlink -f to get realpath, but that’s not available on OS X.

So I built it into Oils (both OSH and YSH):

    source $_this_dir/lib/util.ysh

$_this_dir is a special variable that’s relative to the file it’s being sourced from.

https://www.oilshell.org/release/0.19.0/doc/ref/chap-special...

https://www.oilshell.org/release/latest/doc/ysh-tour.html#ap...

It starts with a _ to let you know it’s magic, similar to _status and _process_sub_status (the latter being unavailable in any shell besides Oils BTW)

So yeah this one reason I say POSIX and shell are “rotting” – they lack extremely basic functionality like this. This could have been done 20 years ago, but there’s nobody really pushing it.

(We may add namespaces to shell, which will change some things in the example above, but it won’t change _this_dir. We can use feedback and help! - See recent story https://lobste.rs/s/h4egaz/oils_winter_status_update )

(copy of lobste.rs comment - https://lobste.rs/s/4emmdl/sh_relative_shell_script_includes...)

Oils Winter Status Update - https://www.oilshell.org/blog/2023/11/status-update.html

What about hard links dude? Symlinks are fine because they're "fake", but it's not like a file must have a single canonical location.

In that case whoever is linking the script is “breaking” it, I’d say. There’s stuff that is simply outside the responsibilities of the script/maintainer. If you start worrying about that for “simple” scripts, the logic for handling that is quickly going to outgrow the initial logic itself.

Besides, hard links don't have a canonical target, they are all equally correct names for the file. So even if the script author wanted to support the same technique for hard links there is no reasonable way to do it.

I think symlinks (or hard links, doesn't matter) are the actual problem here. It is actually completely OK for your code to make certain assumptions about how it's going to be invoked, and otherwise break (loudly, and in an obvious way, if possible).

Creating proper `npm` package that works in many popular variants of J*script environment slowly but surely gets to same level of complexity.

i like how thanks to un*x mentality i can just expect argv to be the most malicious possible implementation where the invoker can put anything in it and it will be there for the invokee, on the os family boasting its support for secure multi user systems:

/usr/sbin/./././some_admin_tool

/usr/sbin/../sbin/../sbin/some_admin_tool

/my_rigged_folder/haha (using symlink)

but i guess it would have required allocating a string or something which may have mattered 40 years ago even though we could have had a better OS in every way 20 years ago

and the same for the process list (in multiple ways, one you can see other people's passwords if they do HISTFILE=/dev/null something --password password, two you can just modify the process list)

and i get the last laugh because un*x lovers will reply to criticism such as "your CLI shit is retarded it logs the passwords to disk" with "NO BRO I USE HISTFILE=/dev/null" to which i reply "that's a lot of shit to type just to not log your password to disk" to which he replies with "NO BRO I USE SPACE PREFIX" and he's still wrong due to previous paragraph. similarily the un*x lover will boast about how he can detect malware / bad guys by using the process list yet nope it can just be bypassed like anything in un*x.

and don't even get me started with how moronic it is for a CLI program to "prompt" the user for a password to avoid this issue (actually you can use patches to make CLI paramters not shared to all users of the system, which is still stupid because none of that type of information should ever be shared in he first place: no other user's process should see any attribute of mine)

and the icing on the cake is the typical way people inside the industry are completely oblivious and say "no man this is actually amazing stuff here"

> It's possible to write extremely powerful tools using shell scripts, and modularize your code. Bourne shell doesn't provide a clear mechanism for referencing relative files. By establishing a command file's base dir, you can reliably reference relative shell scripts.

wow a turing complete language is powerful who knew now if only we had one that wasn't just a cesspit of footguns most of which are there for no real reason and even historically were still invalid

Should use $BASH_SOURCE in place of $0 so it works in files that are sourced.

The incantation at the top of all my scripts, that I learned from other HN folks, is:

  SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

And I guess throw a realpath in there too for symlinks, per TFA.

/bin/sh on FreeBSD isn't bash.

On MacOS, we had interoperability uses between different versions somehow, and some of us could not use `realpath` .

The following worked for everyone:

  SCRIPT_DIR=$(dirname "$(readlink -f "${BASH_SOURCE[0]}")").

You don't need realpath. The `cd` into the directory + `pwd` is equivalent.

BASH_SOURCE[0] obviously only works with bash.

> You don't need realpath

Yes you do (or a suitable alternative, e.g. GNU readlink).

> The `cd` into the directory + `pwd` is equivalent.

You won't know which directory to cd to without realpath. Just stripping off the filename does not get you the correct directory if the file itself is a symlink, which is a common use case.

Doh, you're right. Thank you for the correction.

Reply on news.ycombinator.com