Hacker News Reader

Thanks for sharing! One more curiosity - what PNG compression tool was used and how long did it take to crunch 216GB to 17GB? I'm surprised it didn't crash.

View on HN

I didn't compress an actual image from 216GB to 17GB. Instead, I created the highest resolution image already as a set of tiles (each one 5000x10000 px). Then I tiled them further in two additional zoom levels, until I got 4x8 square tiles of 1250x1250px. From there I worked backwards merging chunks of 2x2 tiles.

The figure 217 GB comes from the cumulative size of all the pandas DataFrames containing the whole dataset (as mgspack files). The lossless compressed figure comes from the total size of the highest zoom level.

It didn't take that long, it was mostly done overnight (so around 10 hours in a i7 server with 24 GB RAM).

Thank you for the detail.

Reply on news.ycombinator.com