Hacker News Reader

Comment by pdonis | Hacker News Reader

> multiprocessing allows concurrency at all and this doesn't

I do not think multiprocessing will meet the concurrency needs of many potential applications.

For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me.

For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.

View on HN

> To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child.

FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme.

That said,

> I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen.

I have as well, and completely agree.

Came here just to agree.

This is nothing but message passing. In the high performance computing world, MPI is the standard scheme for running massively parallel calculations. While Python's pickle may not be the highest performance serialization scheme, it is pretty fast and transparently handles most objects you can throw at it. It certainly does not cover all the same use cases as threads and as you point out shared data it is much easier to use theads.

I ended up having to switch to dill instead of the default pickle a while back in one of my projects, because pickling lambdas is apparently not possible by default.

But in general it's pretty useful.

Same here.

Everything seemed to be working fine until it didn't.

> Every other major language besides Python allows the former

As noted in a reply to your other comment, this is not true -- Ruby has the equivalent of Python's GIL.

Only CRuby has a GIL. JRuby is totally viable for production use and doesn’t have a GIL.

And python also has Jython and other implementations that don't have the GIL. I don't understand what point is being made with that line of thinking?

Does anyone use Jython though? In my experience, only CPython and PyPy are commonly used, both of which have a GIL.

Reply on news.ycombinator.com