Multiprocessing.shared_memory – Shared memory for direct access across processes | Hacker News Reader

Multiprocessing.shared_memory – Shared memory for direct access across processes

docs.python.org

84 pointspjmlp23 comments

Great, the support is still basic, but it's a huge improvement over message passing when you need speed. 2 months ago a student asked me exactly that question: how to share data between processes, and when I told him that with the stdlib he had to manipulate mmap by hand (https://blog.schmichael.com/2011/05/15/sharing-python-data-b...), he was not happy.

Now that asyncio is production ready (we really have had a decent doc and asyncio.run() for a short time), and this, the concurrency story in Python is starting to look decent.

Since the 3.6, things are really going nicely.

Maybe we'll see an async django-like framework emerging from now on instead of the myriad of async flask copycat and django-channels. This would open the gate to live settings, manipulating tasks queues in URL enpoints, having RPC/PUB-SUB baked in. After all, I don't need async to just have exactly the same features as a WSGI framework.

Tom Christie (DRF dev) is working on an async django like toolchain with starlette, databases and uvicorn. https://twitter.com/_tomchristie/status/1100064076191940608

Hopefully django will start supporting asgi soon and let us write async views alongside existing sync stuff.

I like http://trio.rtfd.org way more than asyncio.

Me too, although the scheduling syntax is not to my taste. But trio is not in the stdlib, and is incompatible with most of the current ecosystem. There are talks to integrate it in the future, meanwhile there are compat layers, but it's still in infancy.

For now on, I just use asyncio with a lib to add nurseries to it.

I'm curious what you dont like about the syntax, and which parts of the ecosystem you're waiting on?

"decent doc and asyncio.run() for a short time" -- can you give a URL or some pointers? Thanks.

Starting from 3.7, the documentation is much, much better (https://docs.python.org/3/library/asyncio.html compared to https://docs.python.org/3.6/library/asyncio.html).

As you'll notice, we now can start program with a simple asyncio.run() instead of manually setuping the event loop. Not only it is less verbose and easier, it's also much less error prone. It should have been in asyncio from the begining, but even a long fight on python-ideas didn't work. Fortunatly, Yuri decided to add on his own.

thanks, very helpful

Now combine that with Apache Arrow and you can do some interesting things.

This is great! As someone who codes lots and lots of multiprocessing + queue code I'd love better shared memory support

This rules, but until 3.8 is out I've been using http://semanchuk.com/philip/posix_ipc/ which unfortunately doesn't work on Windows.

Doesn't Apache Arrow solve this problem partially already?

Both 3.7 and 3.8 have so many performance improvements. But still, what people like most about Python is code readability. Performance isn't as important as in the past, because of how hardware performance has scaled.

Very true, but it does let me make stuff faster than my co-workers who are using C++.. without their "well python is so slow" argument from ten years ago.

It is extremely unlikely (I actually wanted to say "never" but many people were bitten for using that word) that Python will be faster than C++.

So if the hope of being fast keeps you with Python, you choose the wrong language.

You can't take advantage of the hardware improvements if you're not using all cores.

Well, with multiprocessing you actually do.

wait, hardware performance hasn't really progressed much in the last 10 year!

Not CPUwise. But they have increased the number of CPUs available and that's where python has problems because of the GIL. Fortunately the multiprocessing library seems like a good workaround to the GIL issue.

Agree, which means that python actually got worse!

I guess we can argue the semantics of "much" but comparing my current ryzen system to my core2duo system from 10 years ago the performance difference is massive.

To be extremely optimistic, performance per clock cycle has doubled. The average clock of server hardware hasn't increased significantly.

The number of cores have increased ridiculously, but that doesn't really help python much.

It does when going old style multi-process.

Which is something that is starting to look interesting again, given the security issues of multi-threading.

Reply on news.ycombinator.com