The fun part was that the way the data got chunked through the pipes was not deterministic, so sometimes you got the desired output from grep, but other times just "binary file matches", even when the raw output from the first tool was identical in both runs. That took quite some head scratching to figure out.
Actually I recently found that coreutils and ls behave fairly well with funny filenames:
Here is an invalid utf-8 byte and then a valid utf-8 sequence
$ x=$'\xce\xce\xbc'
$ touch "$x"
$ ls
?μ
$ ls --escape
\316μ
$ stat *
File: ''$'\316''μ'
-----
Most command line tools are not aware of stuff like this. For example you can touch "x$ANSI_TERMINAL_CODES" and if you do "bash x??" or "python x??", then your terminal will change color because of the escape codes printed back to the terminal.
I just changed Oil to use a well-defined format I called QSN (quoted string notation):
http://www.oilshell.org/blog/2020/04/release-0.8.pre4.html#t...
It adapts Rust's string literal syntax to express arbitrary byte strings precisely and losslessly. (JSON can't express arbitrary byte strings.)
The QSN encoder does UTF-8 decoding with a specific error recovery mechanism. So it's basically like what ls and stat do, but it's more precise.
(If anyone is interested in QSN, please contact me. I think it's more generally useful in a lot of places. It's something we already do but it's precise like JSON.)
% touch foo
% touch 'bar baz'
% ls
'bar baz' foohttps://www.gnu.org/software/coreutils/quotes.html
At least with Gnu, you can recompile your own, non-broken version, which is the only saving grace of these stupid, trendy changes.
I deal with a lot of filenames with spaces and think this change is a great improvement for listing such files. With this change it’s much easier to see where one filename ends and the other begins. Before this change, I had to use the `-1` option to ensure that each filename was listed on a line by itself. Now the filename listings are much more readable and it takes less cognitive effort to take it all in.
The way it handles filenames with ASCII apostrophes/single quotes works particularly well (wraps the filename in double quotes instead of single quotes) and makes it very easy to copy and paste filenames to and from the terminal.
Best of all, this change only applies when standard output is a TTY device so this does not break any shell scripts (even though parsing `ls` is a bad idea in any case) and is still compliant with the POSIX specification[1] which states that “If the output is to a terminal, the format is implementation-defined”.
1. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/l...
the point is that this breaks an unbelievable amount of already deployed scripts. The new functionality should be optional, and accessed via switches and aliases if you like it.
this was very much a change only a few people liked, that they decided to force down literally everyone else's throats. it's very poor stewardship.
I guess I should have figured that oblique references to "ls quoting fiasco" is shorthand for "I don't understand what's wrong but I'm angry about it..."
(On the other hand I would say the grep -a issue is bad both before and after because either way it relies on autodetection. The fundamental issue there is that there is too much variance in encodings, which isn't easy to fix. Luckily UTF-8 is growing in popularity, and it doesn't have this issue because it doesn't require metadata for extremely common operations like "find ascii substring".)
Take it to Reddit. I know perfectly well what's going on, and also know better then to discuss it with people who act like that.
If you’re a human, yes. If you’re a script, it breaks you in half. If you’re a script that has to run on various versions, then maybe it’s time fix yourself and use find. You’re a sophisticated script after all, not one of these who require a human with a debugger. Modern culture may not appreciate that little ‘compat’ thing, but it is essential if you want something to continue to work and not just stop and wait for someone’s educated guesses. Good software doesn’t point fingers at you, it just works. I remember how recently I wanted to check network interfaces on some machine and commanded ‘ifconfig’. Now it’s called ‘ip a’, and there is no ifconfig. I can guess the reason – ifconfig was bad and ip is good. There is also an eternal “FAT” label issue in unetbootin app, which resurrects every time Apple changes its fdisk output format (in every release, as it seems). The workaround is to run it with a cli option – a very thing that unetbootin was created for to skip. This is what makes systems so much fun. Without all these cool things, we would just sit there and cry over our uselessness.
ed: I read below that ls does that in interactive mode only, maybe it’s not that bad then.
I thought I proof-read that sentence when I wrote it...
Can you be more specific? I don't see any commmit or release on that date.
I guess you had this commit in mind:
https://git.savannah.gnu.org/cgit/grep.git/commit/?id=cd36ab...