This is an interesting take on things.
Decompressing using a different algorithm like .bz2 or .xz is noticeably slower, so sticking with gz and shuffling the files might split the difference.
Decompressing using a different algorithm like .bz2 or .xz is noticeably slower, so sticking with gz and shuffling the files might split the difference.
gzip bzip2 xz
1 6.771 24.23 13.251
2 6.581 24.101 12.407
3 6.39 23.955 11.975
4 6.313 24.204 11.801
5 6.153 24.513 11.08
6 6.078 24.768 10.911
7 6.057 23.199 10.781
8 6.033 25.426 10.676
9 6.026 23.486 10.623
That said, xz is in the ballpark and can be significantly smaller.
[1] https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-co...
Also, if you need to use deflate for compatibility, use https://github.com/zlib-ng/zlib-ng , which is substantially faster than either zlib or gzip.
"Arch Linux added support for zstd as a package compression method in October 2019 with the release of the pacman 5.2 package manager, and in January 2020 switched from xz to zstd for the packages in the official repository. Arch uses zstd -c -T0 --ultra -20 -, the size of all compressed packages combined increased by 0.8% (compared to xz), the decompression speed is 1300% faster, decompression memory increased by 50 MiB when using multiple threads, compression memory increases but scales with the number of threads used."
TL;DR zstd is faster, smaller and has a wider range than gzip. For maximum compression, plzip seems to be the best.
[0] https://community.centminmod.com/threads/round-4-compression...