What's the reason for using bz2 here? Wouldn't it be faster to do a one off conversion to zstd? It beats bzip2 in every metric at higher compression levels as far as I know.
That assumes you're processing the data more than once.
Common Crawl delivers the data as bz2. Indeed I store intermediate data in zstd with ZFS.
Is this data available as torrents?
Yeah came here to say a 14% speed up in compression is pretty good!
It's blazingly fast
bzip2 (particularly parallel implementations thereof) are already relatively competitive for compression. The decompression time is where it lags behind because lz77 based algorithms can be incredibly fast at decompression.