Installing extensions in parallel (experimental!)¶
Since EasyBuild v4.5.0 (initial) support is available for installing extensions in parallel, that is to run the installation command for multiple extensions that are ready to install at the same time in the background, to exploit the availability of multiple cores.
Note: this is an experimental feature. Some of the mentioned functionality may be subject to change or be prone to errors.
Contents
Configuration¶
To let EasyBuild install extensions in parallel, the parallel-extensions-install
configuration option must be
enabled.
In addition, since the support for installing extensions in parallel is an experimental feature, the experimental
configuration option must be also be enabled (see also Experimental features).
How many extension installations should be started (at maximum) at the same time is controlled by
the parallel
configuration option.
Requirements¶
For EasyBuild to be able to install a list of extensions in parallel, two requirements must be met:
- The easyblock(s) that are used for installing the extensions must have support for determining
which installed extensions are required to fulfill the dependencies for a particular extension.
This is done via the
required_deps
property method. - EasyBuild must be able to start a single shell command (asynchronously, so it runs in the background)
to perform the installation of the extension via the
run_async
method, and check whether the command has completed via theasync_cmd_check
method.
For an example of how required_deps
, run_async
, and async_cmd_check
can be implemented,
see the RPackage easyblock.
Caveats and Known issues¶
(last update: EasyBuild v4.5.0)
There are a couple of caveats and known issues with installing extensions in parallel, which we hope to resolve in future EasyBuild releases.
This is also why the support for installing extensions in parallel is currently marked as an experimental feature.
Only works for R extensions¶
The support for installing extensions in parallel currently only works for R extensions,
since only the RPackage
easyblock (which is used to install R packages as extensions) implements
the necessary methods (see Requirements).
Supporting other types of extensions (Python, Perl, etc.) requires that the corresponding easyblocks that
are used to install those extensions (like PythonPackage
, PerlModule
) are enhanced to support determining
required dependencies and starting the installation command asynchronously.
List of extensions must be self-contained¶
The EasyBuild framework currently enforces that all required dependencies for a particular extension
are included in the list of extensions that is being installed (specified via the exts_list
easyconfig
parameter). Extensions provided by dependencies are not taken into account (yet).
As a result, trying to install an
R-bundle-Bioconductor
easyconfig with parallel-extensions-install
enabled
will result in an error reporting missing required dependencies.
Skipping of installed extensions is still done sequentially¶
Checking which extensions are already installed for which the installation can be skipped
when --skip
(see Installing additional extensions using -k/-skip) is being used is still done sequentially.
Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.
Sanity check for extensions is still run sequentially¶
The sanity checks that are done for each extension are still done through a sequential loop over the list of extensions.
Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.