Comment by gpderetta | Hacker News Reader

Comment by gpderetta | Hacker News Reader

Parent Full thread

I have another story un the general spirit of doing crazy pointless things with bash.

A couple of jobs ago, I had to write a job to download a large set of files scattered in a deep hierarchy from the web interface of an internal distributed file system. Because the directory page was non standard and didn't contain any url, I couldn't simply use wget or curl in recursive mode, but had to do some light parsing of each tree level.

My simple bash one liner quickly grew in a larger script. Of course, this being bash, it was trivial to make it do the scraping and downloading in parallel, but I needed a way to limit the maximum number of concurrent downloads.

To make a long story short, I ended up reimplementing semaphores in bash on top of message passing via pipes and 'coproc' (which I had just found out).

In retrospect I could have probably implemented something simpler and more robust on top gmake (which interestingly also uses pipes to limit the number of subprocesses but doesn't need a supervisor process).

You can do this with xargs -P (with GNU xargs at least). I often use the pattern:

    do_one_thing() {
      local arg=$1
      ... do stuff with $arg
    }

    do_with_limited_parallelism() {
      cat things-to-do.txt | xargs -P 10 -n 1 -- $0 do_one_thing "$@"
    }

    "$@"

That limits the number of concurrent processes, which is almost always good enough for limiting the number of concurrent downloads. If you have really tiny downloads, then the overhead of starting a connection per process may matter. In that case it's easy to do more than one download per process with say -n 50 instead of -n 1.

xargs -P is great and I have used it often for quick parallelization. The problem I had was that the crawler would recursively invoke itself while crawling the (possibly unbalanced) tree so xargs wouldn't have worked.

OK, I see. Your implementation sounds a little like how "make -j" is actually implemented (in C). This basically sounds like implementing a semaphore with a pipe.

http://make.mad-scientist.net/papers/jobserver-implementatio...

""" Rather than using complex schemes such as a central server, shared memory, sockets, etc. for synchronizing the various instances of make, he suggested we use the simplest, most basic UNIX construct: a pipe. Not only is this easy, but it’s also a relatively portable concept. Most operating systems these days implement some sort of pipe feature, because that paradigm is so useful.

The idea is ingeniously simple: the initial, top-level make creates a pipe and writes N one-byte tokens into the pipe. That pipe is shared between that make and all submakes, and any time any make wants to run a job it first has to read a token from the pipe. Once the job is complete, it writes the token back to the pipe. Since there are only N tokens, you know that you will never invoke more than N jobs. """

Yes, except that gmake implementation is brilliant as it uses the pipe itself as a semaphore, while my version had an explicit process keeping a count and listening to V and P messages.

Unfortunately I didn't know how make -j was implemented at that time.

    mkfifo queue
    run_crawler root > queue &
    parallel -a queue run_crawler > queue

add --group or -k to taste.

Nice. I do need to add parallel to the my set of bash tricks.

So, is it just me that keep refining bash 'one liners' well past the point they should be implemented on a real language?

haha great timing!

This morning I was scraping the images from the 2016 report and given they were like ~230 images I thought it could be cool to download them in parallel, but I quickly realized this would lead to such path and downloaded these in serial.

I still wonder what's the _right way_

It's the same number of bits either way, so the relevant info is were the images are stored and what the bottleneck is in accessing them. If they're all on a single disk drive, for example, it would probably make no difference if you did it in parallel or serial.

Reply on news.ycombinator.com