Comment by fragmede | Hacker News Reader

Comment by fragmede | Hacker News Reader

Parent Full thread

> So:
> cat foobar.img > /dev/sdi

> will stream the file rather than what dd does,

Sorry, that's not quite right. `cat` (and your shell, presumably bash) does the same fundamental thing as `dd`, ie, read block, write block. There's not really an underlying 'stream' primitive that `cat` (or `bash`, as you're using redirection to write the file) is using compared to `dd`

What `cat` does do, is it does a better job of trying to find an optimal blocksize than a naive `dd` call does. `dd` simply defaults to a 512-byte block size, which is just inherited from history when 512-bytes was the alignment for, well, everything.

There are numerous optimizations upon the fundamental read-block, write-block primitives to make it go faster (`cat` makes use of some of these). The linux kernel actually has a "stream file to socket" syscall to avoid the copy to user-land and back to "stream" a file out to the network, but that's not happening here, and there's still reading and writing of blocks in the kernel happening.

See also: http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=blob;f...

That's still not quite right.

Once cat is spawned, the shell gets out of the way, and has nothing to do with it. (Exception: zsh will "helpfully" insert itself with `tee`-like operation in certain situations.)

What `cat` doesn't do is finding an optimal block size for writes. It does a good job of finding optimal block size for reads, but not for writes. Many older block devices perform very poorly when the write block size is not optimal.

The fact that zsh is a parent doesn't necessarily hurt performance. The parent and child can share the stdin fd, and the child could be the only one reading from it.

That's not what I was referring to. Though I should have been more clear: I don't believe that zsh inserts itself in the mentioned situation; just that it does insert itself in some situations, and that makes it an exception to my general "once the process is spawned, the calling shell doesn't matter" statement.

As an example of what I was referring to, in zsh, this

    a >&2 | b

is equivalent to Bourne shell:

    a | tee /dev/stderr | b

except that `tee` is implemented as part of zsh, rather than a separate program (for no difference in performance).

That is, zsh inserted itself into the middle of the pipeline. There are two pipes instead of one; zsh/tee reads from the a pipe, and then writes that data to the b pipe (and stderr). This does hurt performance.

This doesn't seem to be the case for me

    -> % python test.py 2>&1 | python test.py
    COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
    [snip]
    python  24150 someone    0u   CHR  136,0      0t0        3 /dev/pts/0
    python  24150 someone    1w  FIFO    0,8      0t0 72772851 pipe
    python  24150 someone    2w  FIFO    0,8      0t0 72772851 pipe
    python  24150 someone    3w   CHR    5,0      0t0     1049 /dev/tty
    COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
    [snip]
    python  24151 someone    0r  FIFO    0,8      0t0 72772851 pipe
    python  24151 someone    1u   CHR  136,0      0t0        3 /dev/pts/0
    python  24151 someone    2u   CHR  136,0      0t0        3 /dev/pts/0
    python  24151 someone    3w   CHR    5,0      0t0     1049 /dev/tty
    someone@arch-two [01:29:00] [~]
    -> % echo $ZSH_VERSION
    5.2
    someone@arch-two [01:29:16] [~]
    -> % cat test.py
    import os
    import sys
    import subprocess

    with open("/dev/tty", "w") as f:
        subprocess.check_call(["lsof", "-p", str(os.getpid())], stdout=f)

Edit: Ok so I misread your comment and you were talking about >&2. Which is weird because it actually has different behaviour in bash vs zsh. Having said I can't think why you would want the bash behaviour, although the bash behaviour does seem more consistent.

In the particular example I used, the Bourne shell behavior probably has no practical use (but I can write code of no practical use in many languages).

It's not particularly surprising that the zsh behavior is different than bash; zsh redirections are only Bourne shell-like in simple cases.

I dislike zsh's behavior because it isn't what the user typed. It should be simple to look at a command line and see how many pipes it makes; see the processes that will be involved in the pipeline. Zsh's behavior seems to me to be clever and implicit; I'd much rather have dumb and explicit.

Reply on news.ycombinator.com