Hacker News Reader

Comment by Animats | Hacker News Reader

Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload.

"Multiprocessing" is useful when you have a lot of work to do concurrently and not too much data to pass between processes. I've used Python subprocesses that way. Parallelizing your number crunching is probably not going to work very well.

View on HN

See my other replay in this thread.

> Parallelizing your number crunching is probably not going to work very well. [...]

The question is, what exactly does "number crunching" mean? We do aerial imagery analysis, so image processing in essence, which I would classify as a "number crunching" problem. A common thing e.g. is to do a time-series analysis and you can simply start multiple (2, 4, ..., N with clusters, etc.) processes for each problem. Obviously this works because most methods are computation and/or memory heavy - the additional memory requirements and "overhead" of Python itself (IMHO people overestimate the weight of starting new processes instead of threads) is completely dwarfed by the requirements (memory and CPU) of the method itself.

...which is true, but doesn't mean you can just ignore it.

Interpreter state is among the most frequently accessed memory in many applications, meaning it's ideal to have it in cache. The difference between two interpreter states and one might not be big compared to the data being processed, but it's big enough to bump a lot of interpreter state out of cache, which for many programs can have drastic performance implications.

If you don't think cache locality is important, look at radix sort versus quicksort. Radix sort has a much lower O, but performs worse in most cases because of its poor cache locality.

Look, I get that there are fairly easy ways to work around these problems, but let's not just blithely pretend they aren't problems.

Agreed, but there is a lot of misinformation about the topic. I met developers that thought the GIL prevents you from running your program in multiple instances at the same time on one machine - which is obviously not the case.

Sure, it's a problem for specific workloads, and Python will get there eventually - I just don't think it is a deal breaker.

Actually things are not as bad as they used to be. Since 3.4 you can alter the way multiprocessing starts processes:

https://docs.python.org/3/library/multiprocessing.html#conte...

The ``forkserver`` method eliminates most of the problems you mention: child processes are only started once, and they fork() from a totally separate process so they don't inherit all of the resources of the main process (in particular, they don't copy the whole heap). I've found this eliminates 90% of the performance-related issues I used to experience with multiprocessing.

If you're CPU bound (only reason to care about the GIL anyway), then you want one process per core. So at least the L1 memory cache isn't shared. The separate memory consumption is minimal (3-5MB*N cores).

You don't need to do setup/destroy more then once.

If CPU load is an issue, why would you be using an interpreter in the first place?

It seems like people never make this assessment, or use the GIL argument to put interpreted languages down. I personally run into I/O bound problems way more often than CPU bound ones. That said, I'm mainly doing things in the realm of a Python web developer. Scientists probably hit CPU bound problems more often with Python, but seem to drop down to C/C++ extensions without needing to complain about the problems.

A lot of the heavy lifting is done through calls to C libraries anyhow, with Python just being a convenient way to pass the data around.

Indeed, and in that case the GIL is effectively a non-issue (there's no requirement for the GIL to be held by non-python code).

No, no, if you're manipulating Python objects from C code, you have to hold the lock. You can release it only when not doing anything with objects in Python's memory space. Otherwise you get race conditions and intermittent crashes.

> If CPU load is an issue, why would you be using an interpreter in the first place?

You're basically asking why NumPy, SciPy, Numba, etc. even exist.

They exist because Python is ridiculously fast to develop in compared to, say, C++.

By using numpy, etc. you're basically _not_ using the interpreter because you're using C/C++/fortran code that's been compiled with python bindings.

To combine both your points, the best approach (if you like python) is to stick with python due to is ease of development and use libraries such as numpy as far as possible. However, if your use case is CPU bound but not served by those libraries, then you'll either need to develop your own extensions or throw away the interpreter altogether (and go with a different language).

Just because those resources exist does not mean you get to park your 1997 Chevy Cavalier diagonal across three parking spaces.

I'm not going to run your code on my server if your code uses resources so poorly that I can't run other things I want to run on my server.

You are getting downvotes, I suspect, because your comment makes no sense in the context of the post you replied to.

Perhaps you meant to reply to GP?

>Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive

Statically and dynamically loaded binaries are resident in the kernel's page cache. Which while each process will have different locations within its process address space for each process (b/c ALSR), they _should_ be de-duplicated in RAM, ultimately all these seperate in process images will be pointing at the same physical RAM page(s).

So from a hardware cache standpoint you're mostly okay.

That's just the interpreter's executable. All the stuff that's generated from the Python code you load, and any data it generates, is unique to the process.

With Python that's a lot of stuff; I suggest running strace python some_small_script.py to see just how much data Python loads on every single startup.

The other issue with multiprocessing is that it requires the enclosing code to be pickleable, and many Python objects are not pickleable. For example, if I have a thread-safe RPC client and want to send thousands of RPCs using the client, I can't do that with multiprocessing (subprocess pool; threading pools work). RPC clients manage a TCP connection, if you use multiprocess you end up having to make many TCP connections.

Reply on news.ycombinator.com