Mizu – zero-copy parallel R and Python over shared memory | Hacker News Reader

Mizu – zero-copy parallel R and Python over shared memory

github.com

74 pointscscheid11 comments

Amazing. I remember being really disappointed in mirai last year when learning that it couldn’t provide shared access to an in-memory arrow table. Multicore processing in R is great when it works but too often it’s just a matter of time before unwanted and unnecessary data copies completely destroy performance.

I knew mirai wouldn't be the end of the story. Now I consider R to have best-in-class, and I'm excited to share this with the Python community as well.

Thanks for your tireless efforts, can't wait to try it out.

Author here: I wrote mizu (R) and pymizu (Python), packages that offer lock-free async and parallel computing over shared memory, built on the same C core, libmizu. The link above is for the Python package.

A task round trip drops to under one microsecond, from around 100 µs over sockets with Python's ProcessPoolExecutor or mirai, my other R framework. These are numbers from my laptop (an M4 Pro), and benchmark scripts are in the repos for you to run.

Workers claim tasks off lock-free queues, there is no task dispatcher, and system calls are avoided on the hot path.

`PoolExecutor` is a drop-in `concurrent.futures.Executor`, tasks are awaitable in asyncio, NumPy arrays and polars, pyarrow and pandas frames cross as zero-copy views. As mizu and pymizu share the same wire format, data frames and Arrow types also cross zero-copy between worker pools of different languages.

Of course, this is single-machine only as it uses shared memory. The API is still experimental. Usability feedback and bug reports are most welcome at the repos. I'd love to hear about places you'd like to see this used.

Write-up with full benchmarks and the R/Python code examples: https://opensource.posit.co/blog/2026-11-05_mizu-0-1-0/

I have no use for this but I want to commend you on defending your claims and taking objective numeric measurements. The lack of these lately on performance improvement based projects has been disturbing

That does sound... disturbing. I also run benchmarks regularly on both Linux and Mac to ensure not too much divergence between the two.

I won't link it because idk seems toxic but the last one who replied started with "It's pretty tedious to compare" about their slopcode. To be too lazy to ask the LLM to put that in too is incredible

Impressive work! This looks like it could be a game changer for some of our tasks... I don't mean to be one of those pushy people but... does it, perhaps support Polars?

You're in luck! In fact anything that speaks the Arrow C data interface (polars, pyarrow, duckdb).

I had a need for this exact thing a year or so ago - I made a much less feature-full and less generalized version of this. I will definitely be using this going forward, thank you.

You're welcome, and glad you'll find a use for this!

Reply on news.ycombinator.com