Parallel Job Orchestration with GNU Parallel

Many research computing projects involve workloads of large amounts of relatively short, independent computations. Such workloads are a common reason for researchers to move to shared resources. Despite being called an "embarrasingly parallel" problem, orchestrating large amounts of small computational jobs is surprisingly subtle, particularly when dealing with shared resources with their own usage policies and implementations. To deal with this, many tools have been created, used for a while, and then abandoned. The focus of this colloquium is a tool for job orchestration that is very versatile and actively maintained, GNU Parallel, and it is shown how it helps for a number of use cases.
--
This webinar was presented by Ramses van Zon (SciNet) on April 26, 2023, as a part of a series of weekly Compute Ontario Colloquia. The webinar was hosted by SciNet. The colloquia cover different advanced research computing (ARC) and high performance computing (HPC) topics, are 45 to 60 minutes in length, and are delivered by experts in the relevant fields. Further details can be found at www.computeontario.ca/trainin...