Hacker News Reader

Comment by Foxboron | Hacker News Reader

It should be noted that the makepkg.conf file distributed with pacman does not contain the same compression settings as the one used to build official packages.

pacman:

    COMPRESSZST=(zstd -c -z -q -)

https://git.archlinux.org/svntogit/packages.git/tree/trunk/m...

devtools:

    COMPRESSZST=(zstd -c -T0 --ultra -20 -)

https://github.com/archlinux/devtools/blob/master/makepkg-x8...

View on HN

The man page for zstd mentions that using the --ultra flag will cause decompression to take more RAM as well when used to compress. Does this indicate a huge increase in memory to decompress, or just a trivial amount per package, say something large like... `libreoffice-fresh`? Or `go`? They're two of the largest main repo packages I have installed... (followed by linux-firmware)

Without `--ultra`, the decompression memory budget is capped at 8 MB. At `--ultra -20`, it's increased to 32 MB.

That's still less than XZ, which reaches 64 MB.

The respective flag for brotli would be `--large_window 25 --quality 11`

Brotli defines memory use as log2 on command line, i.e., 32 MB = 1 << 25

zstd uses a lookup table where the number given by the user is mapped to a decoding-time memory use. The user just needs to look it up if they want to control decoder memory use.

If one benchmarks zstd with `20` and brotli with `20`, zstd may be using 32 MB of decoding memory, where one is specifying 1 MB for brotli. By default zstd tends to use 8 MB for decoding (however it is variable with encoding effort setting) and brotli 4 MB for decoding (not changing with the encoding effort setting).

Reply on news.ycombinator.com