(1) We have to be able to mount filesystems which have bad filenames and allow users to work with them.
In that situation, that means the kernel cannot reject all requests for bad file names, only creation requests. And perhaps not even that.
It can be a mount option with three values: off; reject all dubious names in all filesystem syscalls; or just in creating syscalls: open with O_CREAT, mknod, link, ...
It could also be a capability. Superuser could work with the dubious filenames and set up capabilities for non-privileged child processes to do same, so people needing to work with a filesystem full of these names can get into an environment where they can do that.
(2) Enforcement at the kernel level is too late in some situations. It doesn't entirely compensate for broken programs. Here is a trivial example of what this means. Suppose that some program receives an argument "foo bar" and decides to split that into "foo" and "bar", treating this as two file names to create or process. Those filenames look fine to the kernel, so it is allowed.
In shell scripts, not all name-related issues arise from files that are already in the system. In fact, shell scripts are often robust in the face of existing files with poor names. For instance names generated with globbing expansion are safe, even if they contain spaces and shell meta-characters. The one danger is that the prefix of the expanded list looks like command line options to the program.
The issues that arise in shell programming from failing to quote variable expansions cannot be fixed outside of the shell. The OS cannot fix the lack of quoting in command $VAR; it can't prevent a shell internal variable VAR from containing spaces.
Actually, you never should, unless you have a very, very good reason. Even without taking into account encoding issues, you still shouldn't.
But you don't need to call the shell to execute external programs.
Or you could use the solutio9n provided by most argument parsers: the "--" option that disables option parsing for the remaining options. Trying to fix it yourself by "fixing" filenames (by prefixing with "./" or whatever) requires you to enumerate all possible problem[1] or you're just creating a new weird machine[2].
You're worrying a lot about specific problems like if a filename might be confused with an option, but there are many other ways filenames can be a problem *even if their names are limited to "simple"[3] alphanumeric characters. If you're accepting input from the user (or worse, over a network), you will have far worse problem than crashing on a bad filename if you aren't strictly validating and parsing that input before sending it to any other program or using it as a filename.
[1] ("#2) Enumerating Badness") http://www.ranum.com/security/computer_security/editorials/d...
[2] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit...
[3] Defining concepts like "alphanumeric" or even "codepoints that can be safely [printed to the screen" are NOT simple co9ncepts if you support Unicode (which most programs should support).
"The "obvious" way to do this is to litter command invocations with "--" before the filename(s). But it turns out this doesn't really work, because not all commands support "--" (ugh!). For example, the widely-used "echo" command is not required to support "--". What's worse, echo does support at least one dash option, so we need to escape leading-dash values somehow. POSIX recommends that you use printf(1) instead of echo(1), but some old systems do not include printf(1)."
So what? If you don't read the documentation for your the programs you run, mistaking filenames for an option is only only one of the many types of problem you might have. The problem of using programs incorrectly cannot be solved by changing filenames. You solve that type of problem by carefully understanding how to run each program you want to use (which means using -- for many (but not all) programs) and strictly validating & sanitizing your input.
If changing filenames makes an incorrect solution work, then it solves it, in that case.
----
In this thread, you're missing the point somewhat - the problem isn't that it's _impossible_ to work with all sorts of filenames using shell, the problem is that it's a PITA to do so, even for experienced users but especially for newbies.
I.e. sure I can use `"$@"` everywhere and quote every single variable I ever use, and prefix every path with `./` or use `--` for every command that supports it. What if I didn't have to? Wouldn't that be better?
I assure you I'm not; I'm trying to explain why worrying about these problems at the filename level doesn't actually solve the problem, which should be handled at a different level of abstraction.
> sure I can use `"$@"` everywhere and quote every single variable
If you are writing Bourne shell without doing that, you are always going to have problems, because arguments are not always filenames. Yes, the syntax can be slightly annoying for historical/compatibility reasons.
> What if I didn't have to?
You don't. Use a different shell, or write your scripts in one of the many languages that are widely available like ruby, python, etc. If a scrip0t has #!/bin/sh at the top, you can probably rewrite it into any language you prefer.
I completely agree that validating input is very important. Indeed, I am recommending that the kernel do input validation on file names.
I agree with others at this thread here: Each program that your program passes a filename to needs to be considered and handled individually.
You cannot rely on "./" being a general solution. What if the program matches the given path against other paths, and does so verbatim without normalization? What if you're explicitly supposed to pass a filename, not a path? In the first case, maybe you claim that the program misbehaves, but one could argue that not having something like "--" is also misbehavior, and in any case, you have to handle it.
Again: Thinking that there is a universal solution, like "./", for a problem that in reality is about inconsistent interfaces and therefore requires you to look at each interface individually will get you into hot waters.
No reason to have relative paths outside the shell for me, relative paths are only a convenience for the shell user that doesn't have to specify the full path as an argument, but if you invoke a program from another program, it doesn't cost you nothing to pass to it as an argument the full path.
That's true, and a very real problem in many cases. A lot of those cases, though, are because people use positional arguments when they should use switched options ('--foo=bar' or '--foo bar') instead. This shields them from all kinds of pretty severe input-validation errors, and not just with filenames. It's a few more characters to type, but a great many option-parsing libs in all sorts of languages can automatically generate terse ('--foo=bar' can be specified as '-f bar')
I know that puts some of the onus on developers, but as you and others have pointed out, relying on the shell to handle these weird cases likely puts responsibility where it doesn't belong (calling subprocesses from another language shouldn't use the shell unless it's overwhelmingly necessary) and on something that handles it poorly (all your points about quoting hell).
That's a narrow quibble, and is exclusively concerned with getting data in to invoked programs intact, from a non-shell invoker. All of your other points about encoding, validation, shell-altering STDOUT, and broken behavior when you are working in the shell (like the whitespace-related issues you point out, which are still a hassle to work around even when using switched options) are well put and very well taken.
The only cases in which I've seen this cause problems are programs which rely on the presence of a tty when they should not, or assume the existence of environment variables that they don't really need.
And it's not like it's all impossible to do yourself without the shell there's just a lot and the shell does it in a nice and well known way. Here are some examples:
Searching the path
Setting custom environment variables
builtins
One weird one that I feel like a lot of people don't know about is that you can run shell scripts from the shell without the shebang. But if you exec one you'll get a format error. It's not good to rely on this but it's another thing that surprises people with messy scripts.
Rust's stdlib doesn't. You must pass arguments as array.
What if yourcommandhere is shell syntax, like "for x in *.foo; do echo $foo; done"? That can't be executed directly as a process argument list.
If all you have is a wrapper for execv, then you have to construct a call to "sh", where that command is all in one argument following a "-c" argument.
You said that Rust doesn't have a library function for calling the shell but only process execution; I'm showing how you make one in such a situation.
A language and its library won't prevent applications from calling the shell, unless it's uselessly crippled.
Then you readdir, loop, println. no need to use the shell
> You said that Rust doesn't have a library function for calling the shell but only process execution; I'm showing how you make one in such a situation.
You're stating the obvious.
My point was that it has no footgun which automatically defers to the shell or does shell-like argument extraction from a single string when spawning child processes. Other languages have those things in their standard library which is a source of errors that rust avoids. It does provide a regular gun which you can repurposed as a make-shift footgun by taking the conscious action of aiming down first.
Yes; you have a Turing complete language, so why would you use the shell?
The point that you've lost sight of here is that someone can do that and someone likely will.
(I don't disagree with you; I think that "shelling out" commands from decent programmign languages is an anti-pattern.)
Have you seen this?
https://doc.rust-lang.org/std/process/struct.Command.html
Look, an example of what I'm talking about, right on the Rust site: how to send "echo hello" to your shell on Windows and Unix.
You don't think someone's going to copy and paste this, and substitute whatever command they want?
> The point that you've lost sight of here is that someone can do that and someone likely will.
Sure, that is inherent in the freedom of powerful tools, you can also use them to do dangerous things. But that is besides the point.
The point is that the default and simplest way is the safe way to spawn commands.
You have to do extra work to invoke dangerous shellisms. You need to at least understand that arguments are conceptually not one long string but an array of several strings, and to invoke a shell you would already have to start with splitting that invocation into an array, so you gain little from then writing the rest of your command as one string, you might as well split up all of them and avoid all those escaping pitfalls.
This is the contrast to other languages which have constructs like system("echo foo"), spawn_sh("echo foo") or whatever which are superficially easier to easier use than launching child processes the safe way. In this regard rust is safer than most other languages, since it has no "construct which calls the shell" - note the active voice used here, "can be used to call a shell", passive, is a different case.
This is all I have claimed in this thread.
- control characters in filenames: irrelevant to non-shell languages
- leading dashes in filenames: doesn't affect normal Python/non-shell languages; for example, unlike in shell, directory walking in Python is implemented by calling os.walk, not execing a separate process "find" and parsing its output
- the lack of a standard character encoding scheme: doesn't affect Python 3, now that the filesystem encoding system is complete and cleanly handles encoding errors
- spaces in filenames can cause problems: irrelevant to non-shell languages
- special metacharacters in filenames cause some problems: irrelevant to non-shell languages
As you admit, these problems are worse in shell (I contend, only meaningfully exist in shell). So why is your article written using shell scripting for its examples?
- If you really believe that shell is worse for this, then you should be warding people away from shell
- If you really believe these problems are pervasive, take on the hard target, instead of going for the easy target of shell
- If these problems really still affect Python programmers who believe they are safe from these footguns that are so prevalent in shell, then you should be warning them, not the shell programmers who already know that shell is an incredibly hard language to write robust programs in
But as your article is using shell for its examples, I can't help but conclude that this is just another piece of shell-zealotry. If you seriously want to fix this problem for people, you should be advising them to stop writing programs in shell. That is undeniably the fastest and easiest mitigation to the majority of these problems. On that, surely you must agree.
-----------------------------------------------------------------
An aside on the topic of leading dashes in filenames: it's true that any program parsing filenames from the command line will run into this issue where data and options can be confused. You diagnose this as an issue with filenames, but I disagree: This is an issue with traditional argument parsing, which is a very poor serialization format. Any number of alternative serialization formats would avoid this issue; for example, https://github.com/NuxiNL/argdata does not have this problem.
I don't understand how this doesn't affect non-shell languages, it's not the shell that interprets these (unless were talking about different things)
> spaces in filenames can cause problems: irrelevant to non-shell languages
Again, this is a format/protocol problem not a language problem. A lot of formats use spaces as delimiters and if you put delimiters in the filename you're going to have to escape it.
I'd say that the issue is that traditional argument parsing uses in-band signalling: a "filename" starting with a hyphen is treated as a control instruction instead of a filename (in some cases, unless a special "double hyphen" control instruction has been seen).
It might be suitable for writing long, complex scripts, but for short, ad-hoc scripts its overhead (both in terms of startup speed and in terms of the boilerplate required to write full-fledged Python scripts compared to simple shell scripts) makes it really unwieldy and awkward to use.
Python still has problems displaying filenames with unprintable characters.
Arguments passed to Python from the shell are still subject to these pitfalls, etc.
See the rest of this thread for more.
They were written in python 2 and 3 by different teams of strong developers at different organizations at different times.
They were universally unmaintainable, unreliable and undebuggable.
Some narrowly avoided installing pip-distributed malware on internal networks.
There is a reason shell is still widely used. I have never worked with anyone that actually knows one of the shell languages that prefers python for shell scripting tasks.
The shell is full of strange and dangerous behaviors, that make even experienced programmers write buggy scripts. The main reason is that the shell wants to be a programming language but works differently from every other programming language, and has a special and complicated syntax.
Also shell scripts are really not that portable at all, if you write a script on a GNU system it's nearly guaranteed that it will not work on a non-GNU system, and vice versa. Not only the shell but more importantly the commands are different, GNU awk is different than standard UNIX awk, GNU sed is different that UNIX sed, grep is different, and so on. Maybe minor difference that produces bug difficult to find and fix.
For me if a shell script becomes more long than 50/100 lines of code you should consider rewriting it in python, shell scripts are only a valid solutions to rapidly write programs that invokes some commands to automate some things, not a valid solution for doing complex tasks.
* Filenames starting with hyphens can get parsed as command line options
* Filenames containing control sequences cause the terminal to misbehave if they are printed to the screen
* Filenames containing newlines can't be easily included in all sorts of line-by-line file formats.
Finally, as the article also points out, the shell is not going away anytime soon. Getting rid of it is not the easy way out.
Python 3 is full of problems with fsencoding, to the point that some people want to deprecate the plain C locale (!).
This is hardly a solution.
Another problem is that doing many things that would be simple in the shell require a lot of boilerplate in Python. That kind of boilerplate might be worth putting up with for large programs, but for something that could be easily handled with a short shell script it's really annoying.
The chances of getting the kernel to outright ban some of the more dangerous filenames seem a lot higher than simply eliminating sh/bash...
https://github.com/david-a-wheeler/linux/blob/squelch-1/secu...