Hacker News Reader

Comment by rdtsc | Hacker News Reader

There is a nice hack for multiprocessing module to use for IO-bound code without having to fork processes:

   from multiprocessing.dummy import Pool as ThreadPool
   pool = ThreadPool(16)
   res = pool.map(one_arg_fun, [arg1, arg2, ...])

A 3 line IO parallelism speedup trick. Used it fetch stuff from multiple servers recently.

View on HN

I would recommend you use coroutines rather than multiprocessing.dummy for balancing blocking operations (like IO). I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1].

Python3 has asyncio as part of the stdlib but for Python2 there are a variety of coroutine libraries out there (I think the most widely recognized being gevent).

[1] https://www.youtube.com/watch?v=Obt-vMVdM8s

Switching code-base to Python 3 or installing gevent/eventlet is certainly more than a 3 line hack.

> I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1].

Yap. But I am fetching and waiting for data to come from servers half-way across the country. Not worried about too much thread switching overhead. The point it was still faster than a for loop with a fetch. It was replaced by 3 lines which was 10x faster.

I'm not following. Can you explain the hack? Python releases the GIL during I/O right, isn't that desirable?

> Python releases the GIL during I/O right, isn't that desirable?

Python releases GIL during IO so that's why thread-based pool works for IO parallelism and there is no need to fork. Was that what you mentioned? Sorry, I don't think I completely followed.

I just re-read your comment again and the one above it. I was confusing MT and MP. Thanks.

can you elaborate?

It's the equivalent of: (edit: actually an improvement over)

    from threading import Thread


    def blocking_function(arg):
        ....


    pool = []
    for arg in args[:16]:
        f = lambda: blocking_function(arg)
        thread = Thread(target=f)
        thread.start()
        pool.append(thread)

Edit: see below, this doesn't include job queueing but rather, hard limits your input to maximum 16 args. The point is, the code is a nice, easy way to start a number of threads working on a list of arguments.

Not quite. You can submit as many jobs as you'd like to the pool and it will delegate them to 16 threads. The pool maintains a queue of work to be done and the fixed number of threads consume from the queue.

Almost but as quinnftw mentioned it does a bit more, specifically it limits the number of threads it runs to 16. I can pass 1000 url or arguments into it and it won't spawn 1000 threads but keep it at 16 max. Also I think it does a bit a better job with exception propagation.

Multiprocessing offers a ThreadPool for multi-threading..

It doesn't have true CPU parallelism like multiple processes would have, but it works for IO code.

It's a little gold nuggest in the multiprocessing lib.

Reply on news.ycombinator.com