GNU parallel (gnu.org/software/parallel/) is a powerful tool for evoking multiple processes. For example:

ls *.log | parallel -j8 xz -v9z

Will execute 8 xz processes in parallel to compress all the files with .log extension. By default, it will block until all the child processes return. You can also use it as the semaphore.

for i in *.log ; do
echo $i
parallel --semaphore -j8 gzip $i ";" echo done
done
parallel --semaphore --wait

Follow

@meisam For the second example, you may also:

ls *.log | parallel -j8 'echo {}; gzip {}; echo done'

@ashwinvis My focus was on using the GNU parallel for running stuff in parallel not using xz/gzip per se.

The second example is just there to show how GNU parallel can be used as semaphore. There are many other examples on the things you can do on their website: gnu.org/software/parallel/para

BTW, if the compression ratio is the critical factor in choosing an algorithm, zstd’s performance is nowhere near xz. Here’s a sample benchmark: community.centminmod.com/threa

@meisam Sure. But the speed of compression / and uncompression is a good tradeoff.

@ashwinvis Not necessarily for everybody. Sometimes I have a hard-limit on the total size of my data and lots of processing power to spare. (first world problems :oh_no_blob: )

Sign in to participate in the conversation
FediScience.org

Fediscience is the social network for scientists.