Hacker News Reader

Comment by mvgoogler | Hacker News Reader

Except that if you read the problem statement, you'll see that tee doesn't meet the requirement of 'taking advantage of all available I/O parallelism'. Tee writes to each output file sequentially.

Given that their core business is data-stores that take full advantage of emerging storage technologies (such as flash), this seems like pretty relevant question.

View on HN

Except that if you do: whatever | tee filea | tee fileb | tee filec | tee filed >/dev/null, you'll get parallelism. This works since tee will write first to stdout, and then to the file you pass it. This may not be "all available" parallelism, but I suspect that it's a decent approximation of "good enough".

I did read the problem statement, but I thought tee was writing to all files in parallel. It's surprising that there seems to be no way to achieve that.

Tee writes in one-thread-per-file level of parallelism; simple experiment:

  $ tee foo.bin bar.bin baz.bin < /dev/zero

writes to all three files `at once' (in small time-slices, actually). Used GNU tee from coreutils-8.10.

Perhaps OP meant several-threads-per-file level of parallelism, where number_threads > number_files?

tee has a simple main loop:

  while not EOF: 
    (synchrounously) read up to BUFSIZE bytes from stdin into buf 
    for each output file: 
      (synchrounously) write buf to file

(source code here: http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tee....)

All of the reads and writes are done through synchronous file apis. I don't see where you are getting one-thread-per-file (or any other kind of parallelism) there.

Reply on news.ycombinator.com