This situation reminds me a little bit of trying to work with very large images in ImageMagick, like 60k x 60k resolution. A simple resize was taking 6 hours on my mac 10 years ago, due to IM trying to allocate the whole image at once and then swapping non-stop. And then I discovered that the Graphics Magick fork did streamed resizing, and did the task in a couple of minutes. It’s a small and relatively easy change, but someone had to prioritize handling large inputs first. This is one reason software is hard for me; every time I need to process large inputs I wish I had written a streaming mechanism, but every time I start a project I decide to do the easy thing first and wait until I critically need streaming. Maybe it’s the right thing to do to avoid over-engineering, but this comes up often enough that I’m usually in a mild state of frustration about something being under-developed.