GNU parallel (gnu.org/software/parallel/) is a powerful tool for evoking multiple processes. For example:

ls *.log | parallel -j8 xz -v9z

Will execute 8 xz processes in parallel to compress all the files with .log extension. By default, it will block until all the child processes return. You can also use it as the semaphore.

for i in *.log ; do
echo $i
parallel --semaphore -j8 gzip $i ";" echo done
done
parallel --semaphore --wait

Follow

@meisam All good, but if you can, try to use zstd :D. I used it once and never looked back.

Sign in to participate in the conversation
FediScience.org

Fediscience is the social network for scientists.