Zip format also defaults threads to number of CPUs, but only hit 30% CPU (over 16 threads) on some test data.
Zip format also defaults threads to number of CPUs, but only hit 30% CPU (over 16 threads) on some test data.
With command line, I arrived eventually to a set of switches to use, but the -mt option seems to allocate core per file. That means, that if the filesizes of the archived files are not somewhat distributed, it won't work very well. The extreme is, if you have one huge file and few small ones, the small ones will be compressed quickly and you will be still waiting for the huge one to finish with a single core.
And then there are apps that use liblzma underneath. `flatpak build-bundle` can take almost forever.
The p7zip version by default uses all cores and hits 100% on all of them compressing a single file. Perhaps p7zip or my OS has a different set of default options?
this can't be true in solid archive mode, which should almost always be used (otherwise you might as well use zip). also:
$ tee {1..10} <<< 1
1
$ seq 1 1000000 > 11
$ time 7z a x.7z {1..11}; rm x.7z
7-Zip (z) 21.07 (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-12-26
64-bit locale=en_US.UTF-8 Threads:12, ASM
Scanning the drive:
11 files, 78888917 bytes (76 MiB)
Creating archive: x.7z
Add new data to archive: 11 files, 78888917 bytes (76 MiB)
Files read from disk: 11
Archive size: 2680630 bytes (2618 KiB)
Everything is Ok
7z a x.7z {1..11} 19.89s user 0.09s system 205% cpu 9.733 total
$ time 7z a -mmt=1 x.7z {1..11}; rm x.7z
7-Zip (z) 21.07 (x64) : Copyright (c) 1999-2021 Igor Pavlov : 2021-12-26
64-bit locale=en_US.UTF-8 Threads:12, ASM
Scanning the drive:
11 files, 78888917 bytes (76 MiB)
Creating archive: x.7z
Add new data to archive: 11 files, 78888917 bytes (76 MiB)
Files read from disk: 11
Archive size: 2779833 bytes (2715 KiB)
Everything is Ok
7z a -mmt=1 x.7z {1..11} 21.88s user 0.05s system 99% cpu 21.943 total $ time 7z a x.7z ubuntu-20.04.4-desktop-amd64.iso
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,24 CPUs AMD Ryzen Threadripper 2920X 12-Core Processor (800F82),ASM,AES-NI)
Scanning the drive:
1 file, 3379068928 bytes (3223 MiB)
Creating archive: x.7z
Items to compress: 1
Files read from disk: 1
Archive size: 3277419065 bytes (3126 MiB)
Everything is Ok
real 1m23,295s
user 25m24,447s
sys 0m27,991s
is quite a difference compared to: $ time 7z a -t7z -m0=lzma x.7z ubuntu-20.04.4-desktop-amd64.iso
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,24 CPUs AMD Ryzen Threadripper 2920X 12-Core Processor (800F82),ASM,AES-NI)
Scanning the drive:
1 file, 3379068928 bytes (3223 MiB)
Creating archive: x.7z
Items to compress: 1
Files read from disk: 1
Archive size: 3305891811 bytes (3153 MiB)
Everything is Ok
real 7m5,647s
user 13m9,589s
sys 0m6,757s