Hacker News Reader

Comment by brmgb | Hacker News Reader

The quotes are only used outside of pipes. One file per line without quote has always been the comportment of ls when piped (but with shell globbing you rarely if ever have to pipe ls anyway).

View on HN

Or rather, a newline after each filename. However, as filenames themselves can contain newlines, a single name can span several lines. These new fancy quoting styles actually finally make it possible to parse the output of ls (although I still can't think of a reason to actually do so).

Except filenames can contain quotes as well as newlines, so it is still not possible to parse the output of ls[1][2]. In fact, filenames can contain any byte except for NUL (the terminator) and forward slash (the directory separator). Dealing with this reliably means using NUL separators everywhere, as in `find -print0`, `grep --null`, or any of a shedload of similar flags. Unfortunately POSIX doesn't seem to support any of these, and it's unlikely that NUL support will be universally available everywhere (let alone implemented using the same flags).

[1] https://mywiki.wooledge.org/ParsingLs

[2] https://unix.stackexchange.com/q/128985/3645

Quotes are a non-issue. The classic problem is that we can't tell apart newlines that are part of filenames, and newlines that separate filenames; however, recent coreutils ls provides an alternate format where this is not an issue.

It's not pretty, and it shouldn't be done but it's doable (in many shells, but none of this is specified in POSIX):

  ls --quoting-style=shell-escape-always | while read -r escaped; do
    eval "filename=$escaped"
    echo "'$filename'"
  done

Doesn't work:

  $ cd "$(mktemp --directory)"
  $ touch $'foo\'\n\'bar'
  $ ls --quoting-style=shell-escape-always | while read -r escaped; do
  >   eval "filename=$escaped"
  >   echo "'$filename'"
  > done
  'foo'
  'bar'

There's no way to tell that that's one file rather than two. Seriously, parsing ls output reliably isn't possible. I've seen exactly one example of a legitimate ls in a script (just this week, for the first time in many years using Bash), and all that did was to check something like whether a directory was empty.

Are you sure it didn't work? How does the output look if you change the single quotes in "'$filename'" to some other character? (I mean, the point of this snippet is to parse the output of ls, i.e. assign each path to $filename, once, and then use it in some way. Of course the newline will still be there if you just echo the name.)

As per my original comment, literally any separator other than NUL (or slash if you're only ever dealing with filenames, not paths, even though that would be hella confusing) breaks with some filenames. Just use NUL.

With the given options, ls is no longer trying to separate "raw" filenames with any character. Rather, it encodes them unambiguously, and hence we can parse the newline-separated output just fine. You can see the same "trick" used in one of the links of your first comment (the one that warns against parsing ls output).

The example produces ambiguous output. But you can count the number of times echo was called to see that it worked.

    $ touch $'foo\'\n\'bar'
    $ ls --quoting-style=shell-escape-always | while read -r escaped; do
    >   eval "filename=$escaped"
    >   echo "[$filename]"
    > done
    [foo'
    'bar]

And this is why I find the idea of Powershell to be intriguing.

Reply on news.ycombinator.com