Coreutils Gotchas (2015) | Hacker News Reader

Coreutils Gotchas (2015)

pixelbeat.org

103 pointstosh41 comments

Somewhat relevant: the default sed that ships on all macos is freebsd sed.

The basic featureset and syntax is about the same, but the more advanced parameters and features have different syntax (or not implemented) when compared to gnu sed.

This caused me quite a headache in the past figuring out why certain scripts works (or breaks!) on local machine but not in others; I started taking closer look on all tool versions since then.

(The solution is to just use gnu sed with https://formulae.brew.sh/formula/gnu-sed)

Similar for other tools. "Awk" on MacOS is mawk, which is significantly different from the gawk that Linux uses as "awk".

Edit: As mentioned below, awk is mawk for some Linux distros, gawk for others, and something else on still others :)

Which linux are you referring to? I'm almost certain than Ubuntu 16.04 and 18.04 both ship with mawk as the default.

Edit: just checked and my vps than runs 20.10 uses gawk

Edit 2: 18.04 also comes with gawk, at least the install I have. I know I have worked with mawk on a linux machine- could it have been Ubuntu 16.04?

Ah...yeah, I was probably over generalizing when I said that, though it's probably fair to say that gawk is the default awk for many popular Linux distros.

Debian & Ubuntu have moved back and forth between using mawk or gawk several times.

Debian Jessie (circa 2016) used gawk.

Debian Stretch (circa 2017) used mawk.

Today, Debian uses gawk.

Have you flipped the final one? Both my Buster and Bullseye installation only have mawk.

Trying to dig in deeper makes my head explode though. mawk is marked required yet with update-alternatives priority of 5, and gawk is marked optional but with a priority 10. I guess the intention is that mawk is good enough for base installs, but you probably want gawk.

It's definitely hard to track down. Mawk is often way faster than gawk, but gawk has more features (sockets even...crazy) and maybe fewer arbitrary limits.

IIRC (no Ubuntu VM handy right now) mawk is the default with a minimal LTS install, using alternatives to point awk at it. When you apt install gawk it has a higher priority in alternatives and flips the awk symlinks over to gawk automagically.

> gawk it has a higher priority in alternatives and flips the awk symlinks over to gawk automagically

That must be what happened. I just checked my other machine that had a pretty new install of 20.04 on it, and it does have mawk. I'm happy to see I'm not crazy.

True, although sometimes mawk is installed, as others mention.

If you need gawk, put gawk in your shebang, and if nothing else the end user has a chance at understanding why they’re seeing a crash. Also it will crash on line 1, nice and loud.

See https://stackoverflow.com/q/24275070 for a list of differences between various sed versions. Just seeing the length of the answer is intimidating.

Perl is another great option for portability.

Another plus of Perl is that it’s a little easier to specify tabs new lines and other escaped characters than in sed in my experience. In Perl I’ll usually get it right the first time, but in sed I might have to do something dumb like sticking a literal newline in a variable or something.

There's a bunch of tools that trip me up between the BSD and GNU versions.

Find is the big one - so much so that i "cargo install fd-find" on my boxes these days and use the rust version across all hosts.

I briefly flirted with using zsh's enhanced glob abilities, e.g.

  find . -type f -size +1m -ls

vs.

  ls **/*(.Lm+1)

The biggest problem with this is using the -i flag ("in place") which expects an argument on BSD/MacOS but not on GNU/Linux.

"The solution is to just use gnu sed with https://formulae.brew.sh/formula/gnu-sed)"

Problem for me is I use more than just MacOS and Linux. I would have install GNU sed on every OS I use. I make small OS images for booting from USB. I use a non-busybox multi-call binary as the initial userland to conserve space, part of filesystem embedded in the kernel (sort of like initrd on Linux I guess). Early in the boot and setup process I use sed in scripts. Thus I would have to make sure there was a copy of GNU sed installed, preferably added to the multi-call binary. Not sure the work is really worth it just for a few convenience features. I can do anything I need to do with BSD sed just as well as with GNU sed.

Instead, the solution I chose is to write sed scripts for NetBSD's sed. These work on Linux, MacOS, FreeBSD, OpenBSD, Plan9, etc., old or new. There is one feature it has that GNU does not: "-a"; it can be used to do true (but limited) "in-place editing" without creating a temp file. Not sure why but NetBSD has since changed their sed to be more FreeBSD and GNU-compatible, e.g., adding "-i" and "-g/-G", to enable automatic temp file creation then removal^1 and GNU regex, respectively. However if we start using these "features" everywhere, then our scripts may not work with older userlands.

1. This is often called "in-place" editing, but if one examines the source code one will see there is still a temp file (tmpfname) created. We are being spared the step of manually creating then removing a temp file, but we still need the requisite filesystem space available for one.

Is there a way to automatically set dd's ibs= and obs= depending on the block and/or stripe sizes of the disks involved? It's weird that optimizing a large transfer involves manually looking up and then setting parameters that the computer itself already knows.

(Though I suppose it only "knows" them through proprietary APIs that a program like dd might not want to build into itself. Still, that's what wrapper scripts are for, no?)

Speaking of, is this something sendfile(2) thinks about? Trying to optimize between time spent reading blocks from the disk and time spent writing packets to the NIC?

A lot of use cases of `dd` are better served by `head -c $bytes`. `dd` does provide a lot more control if you need it, but when you don't just use head.

I don't know what you're using dd(1) for where `head -c` would serve (reading the MBR off a disk?) Seems kind of a niche use of dd(1) to me, compared to e.g. mirroring one disk to another disk.

Personally, my own favorite use of dd(1) that I've stumbled upon recently (when making backups of database servers) is that dd(1) functions as a (many times) faster cp(1), when copying single large files between disks with large stripe sizes (e.g. when one or both volumes is backed by a RAID0ed NVMe pools with stripe≥256MiB.)

I think dd(1) is still below optimal throughput here, though, because it's not taking advantage of the huge number of IOPS these disks can put out. (I.e. it's not trying to saturate the IO queue by reading+writing the file's blocks as parallel operations.) Is there a tool that does do that? Like one of the many "parallel rsync" tools, but for single-file use? (Basically, BitTorrent, but for disk:disk rather than disk:network?)

(For my original use case, backing up a database, I had lots of individual 1GB heap files, so GNU parallel + dd(1) actually was saturating the IOPS capacity of the src+dest disk pools. But ever-so-slightly-different use-cases, e.g. moving a tarball of said database around, wouldn't be so amenable.)

The better answer to your question is that you don't /want/ dd to use the filesystem block size because this isn't optimal.

Generally speaking, the larger bs= you use, the faster things will be as it minimizes the syscall overhead. Don't worry about the particulars of the underlying filesystem or block device. Just set the buffers as big as you can reasonably afford given available memory.

And yes, sendfile or io_uring are designed to avoid this.

That supposes a fast block device (i.e. one that can accept/queue reads/writes in its controller faster than you can context switch to the kernel to submit those reads/writes.)

For a slow block device, with no controller-side queue/buffering — say, a floppy disk, the original use-case for dd(1) — you really want to try to ensure that each read(2) or write(2) is for exactly a single disk sector; otherwise you'll end up suffering from read/write amplification which could give you a 2x-or-more penalty on your copy. (Or you rely on the kernel to buffer and coalesce your writes. But that presupposes you're writing to a filesystem, not directly to a disk.)

(Now that I think about it, I'm kind of surprised dd(1) doesn't default to doing "adaptive" block sizes, like TCP's window size — starting low and adjusting up until it finds the optimum, and adjusting back down if it ever notices writes doing patterned straddling slowdowns representing IO "packets" getting split.)

I really want to add this bit:

    cut -d delimiter -f fields
    sort -t delimiter -k fields
    awk -F delimiter

Padraig's a nice guy, why not email him!

>cut doesn't work with fields separated by arbitrary whitespace. It's often better to use awk

I was working on a script to provide cut like syntax using awk, mainly for awk's default field separation and regexp delimiters. It isn't ready, but I just created a repo with what I have so far: https://github.com/learnbyexample/regexp-cut

Nice tool. I just found that if you have fields separated by arbitrary whitespaces, piping that through xargs trims multiple whitespaces into a single whitespace.

```$ echo "words are far apart" | xargs```

Although, it does have its caveats. https://stackoverflow.com/a/12973694

>cut doesn't work with fields separated by arbitrary whitespace. It's often better to use awk

I use `cut` precisely when I want this behaviour. If something actually is tab-separated, I don't want `cut -f3` to give me the 4th column just because the value in column 1 was the empty string.

Over 30 years of using GNU chmod, and only today did I learn about "chmod -R a-x,a+X" to avoid constantly doing two "find | chmod" commands.

This reminds me of the time I tried to replace a region in an image I created with dd and forgot to add the `notrunc` option. Which resulted in the image file being truncated and instead of the few hundred gigabytes only a few megabytes were left.

some previous discussion:

6 years ago https://news.ycombinator.com/item?id=10648255

Worth considering that shells might have builtins for things like kill, echo, test, time, and others. As an example, bash has its own builtin kill command and it isn't documented by the kill(1) man page.

> echo is non portable and its behaviour diverges between systems and shell builtins

`echo` is portable: https://pubs.opengroup.org/onlinepubs/009695399/utilities/ec... Any options passed to `echo` are not. Although I agree `printf` should be used instead.

> Any options passed to `echo` are not

Using common sense, this is probably what they mean by 'non portable' isn't it?

It probably is what they meant. However, what was written is that `echo` itself is non-portable, which isn't correct.

The context of the article is the Gnu coreutils. The coreutils version of echo accepts options which is not allowed according to the link you referenced. Therefore within the context of the article stating echo is non-portable is correct.

GNU coreutils isn't a standard. It conforms to and extends the POSIX standard. If it was just talking about GNU echo, then that `echo` would work the same on every machine. It even explicitly mentions other `echo` implementations, like the shell builtins. `echo` itself is portable, none of its flags are.

You seem to have a very narrow definition of "portable." Can you link to the definition of the jargon term you appear to be using here?

Also, have you considered that some words have colloquial meanings too, and that it doesn't make sense to label them as "not correct"?

> You seem to have a very narrow definition of "portable."

Can you explain how wanting to include more programs into the definition of portable is narrow?

> Can you link to the definition of the jargon term you appear to be using here?

The same definition I've always seen in every community discussing shell-scripting: if it's specified by the POSIX (Portable Operating System Interface) standard. POSIX does have a definition of `echo`. GNU echo does not conform to it by default, but can be configured to.

> Also, have you considered that some words have colloquial meanings too, and that it doesn't make sense to label them as "not correct"?

But it makes sense to label `echo` itself as not portable? The argument can be made that `GNU echo` is not portable (even though it can be with an environment variable), however that should be specified, instead of mentioning there are several implementations where it is non-standard.

You sound like a zealot to be honest. You misconstrued my post and gave a disingenuous response IMO. And you didn't answer my questions.

Doesn't that make echo non-portable in practice.

`echo -e` should print "-e\n" according to POSIX but the most common implementation will print "". As far as I can tell there is no way to work around this.

It makes relying on meaningful options passed to `echo` non-portable. You can always rely on `echo` being on the system (thanks to POSIX), which is what makes it portable. Relying on the behavior of -e/-n/etc is not.

But that is my point. Print the string "-e\n" in a portable way. I can't think of a way. For GNU coreutils you would need to do `echo - -e` but on a complaint echo you would need to do `echo -e`. So maybe it is portable for some strings. But it can't be used for all cases. Even something as simple as outputting a user entered string can't be done because you don't know if it is handled specially.

You can with the POSIXLY_CORRECT environment variable:

$ /bin/echo --version echo (GNU coreutils) 8.30 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Written by Brian Fox and Chet Ramey. $ POSIXLY_CORRECT=1 /bin/echo -e -e

Which is why I say relying on `echo` is fine, since its standard. Relying on `echo` to use options is not. That's also why the author is correct that you should use `printf`, because POSIXLY_CORRECT is not set, then you can run into non-standard behavior, like you showed.

Reply on news.ycombinator.com