@ashwinvis My focus was on using the GNU parallel for running stuff in parallel not using xz/gzip per se.
The second example is just there to show how GNU parallel can be used as semaphore. There are many other examples on the things you can do on their website: https://www.gnu.org/software/parallel/parallel.html#EXAMPLE:-Working-as-xargs--n1.-Argument-appending
BTW, if the compression ratio is the critical factor in choosing an algorithm, zstd’s performance is nowhere near xz. Here’s a sample benchmark: https://community.centminmod.com/threads/compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.12764/
@meisam Sure. But the speed of compression / and uncompression is a good tradeoff.
@ashwinvis Not necessarily for everybody. Sometimes I have a hard-limit on the total size of my data and lots of processing power to spare. (first world problems )
Fediscience is the social network for scientists.