easybuild.framework.easyblock module

Generic EasyBuild support for building and installing software. The EasyBlock class should serve as a base class for all easyblocks.

Authors:

• Stijn De Weirdt (Ghent University)
• Dries Verdegem (Ghent University)
• Kenneth Hoste (Ghent University)
• Pieter De Baets (Ghent University)
• Jens Timmerman (Ghent University)
• Toon Willems (Ghent University)
• Ward Poelmans (Ghent University)
• Fotis Georgatos (Uni.Lu, NTUA)
• Damian Alvarez (Forschungszentrum Juelich GmbH)
• Maxime Boissonneault (Compute Canada)
• Davide Vanzo (Vanderbilt University)
• Caspar van Leeuwen (SURF)

class easybuild.framework.easyblock.EasyBlock(ec)

Bases: object

Generic support for building and installing software, base class for actual easyblocks.

apply_post_install_patches(patches=None)

Apply post-install patch files that are specified via the ‘postinstallpatches’ easyconfig parameter.

banned_linked_shared_libs()

List of shared libraries which are not allowed to be linked in any installed binary/library. Supported values are pure library names without ‘lib’ prefix or extension (‘example’), file names (‘libexample.so’), and full paths (‘/usr/lib64/libexample.so’).

bin_lib_subdirs()

List of subdirectories for binaries and libraries for this software installation. This is used during the sanity check to check RPATH linking and banned/required linked shared libraries.

build_step()

Build software (abstract method).

check_accepted_eula(name=None, more_info=None)

Check whether EULA for this software is accepted in current EasyBuild configuration.

check_checksums()

Check whether a SHA256 checksum is available for all sources & patches (incl. extensions).

Returns:list of strings describing checksum issues (missing checksums, wrong checksum type, etc.)

check_checksums_for(ent, sub='', source_cnt=None)

Utility method: check whether SHA256 checksums for all sources/patches are available, for given entity

check_readiness_step()

Verify if all is ok to start build.

checksum_step()

Verify checksum of sources and patches, if a checksum is available.

clean_up_fake_module(fake_mod_data)

Clean up fake module.

cleanup_step()

Cleanup leftover mess: remove/clean build directory

except when we’re building in the installation directory or cleanup_builddir is False, otherwise we remove the installation

close_log()

Shutdown the logger.

collect_exts_file_info(fetch_files=True, verify_checksums=True)

Collect information on source and patch files for extensions.

Parameters:

• fetch_files – whether or not to fetch files (if False, path to files will be missing from info)
• verify_checksums – whether or not to verify checksums

Returns:

list of dict values, one per extension, with information on source/patch files.

configure_step()

Configure build (abstract method).

det_iter_cnt()

Determine iteration count based on configure/build/install options that may be lists.

dry_run_msg(msg, *args)

Print dry run message.

extensions_step(fetch=False, install=True)

After make install, run this. - only if variable len(exts_list) > 0 - optionally: load module that was just created using temp module file - find source for extensions, in ‘extensions’ (and ‘packages’ for legacy reasons) - run extra_extensions

static extra_options(extra=None)

Extra options method which will be passed to the EasyConfig constructor.

extract_step()

Unpack the source files.

fetch_extension_sources(skip_checksums=False)

Fetch source and patch files for extensions (DEPRECATED, use collect_exts_file_info instead).

fetch_patches(patch_specs=None, extension=False, checksums=None)

Add a list of patches. All patches will be checked if a file exists (or can be located)

fetch_source(source, checksum=None, extension=False, download_instructions=None)

Get a specific source (tarball, iso, url) Will be tested for existence or can be located

Parameters:

• source – source to be found (single dictionary in ‘sources’ list, or filename)
• checksum – checksum corresponding to source
• extension – flag if being called from collect_exts_file_info()

fetch_sources(sources=None, checksums=None)

Add a list of source files (can be tarballs, isos, urls). All source files will be checked if a file exists (or can be located)

Parameters:

• sources – list of sources to fetch (if None, use ‘sources’ easyconfig parameter)
• checksums – list of checksums for sources

fetch_step(skip_checksums=False)

Fetch source files and patches (incl. extensions).

fix_shebang()

Fix shebang lines for specified files.

full_mod_name

Full module name (including subdirectory in module install path)

gen_builddir()

Generate the (unique) name for the builddir

gen_installdir()

Generate the name of the installation directory.

get_checksum_for(checksums, filename=None, index=None)

Obtain checksum for given filename.

Parameters:

• checksums – a list or tuple of checksums (or None)
• filename – name of the file to obtain checksum for
• index – index of file in list

get_checksums_from_json(always_read=False)

Get checksums for this software that are provided in a checksums.json file

Parameters:always_read – always read the checksums.json file, even if it has been read before

static get_steps(run_test_cases=True, iteration_count=1)

Return a list of all steps to be performed.

guess_start_dir()

Return the directory where to start the whole configure/make/make install cycle from - typically self.src[0][‘finalpath’] - start_dir option – if abspath: use that – else, treat it as subdir for regular procedure

handle_iterate_opts()

Handle options relevant during iterated part of build/install procedure.

init_dry_run()

Initialise easyblock instance for performing a dry run.

init_ext_instances()

Create class instances for all extensions.

install_extensions(install=True)

Install extensions.

Parameters:install – actually install extensions, don’t just prepare environment for installing

install_extensions_parallel(install=True)

Install extensions in parallel.

Parameters:install – actually install extensions, don’t just prepare environment for installing

install_extensions_sequential(install=True)

Install extensions sequentially.

Parameters:install – actually install extensions, don’t just prepare environment for installing

install_step()

Install built software (abstract method).

invalidate_module_caches(modpath)

Helper method to invalidate module caches for specified module path.

load_dependency_modules()

Load dependency modules.

load_fake_module(purge=False, extra_modules=None, verbose=False)

Create and load fake module.

Parameters:

• purge – boolean indicating whether or not to purge currently loaded modules first
• extra_modules – list of extra modules to load (these are loaded before loading the ‘self’ module)

load_module(mod_paths=None, purge=True, extra_modules=None, verbose=True)

Load module for this software package/version, after purging all currently loaded modules.

Parameters:

• mod_paths – list of (additional) module paths to take into account
• purge – boolean indicating whether or not to purge currently loaded modules first
• extra_modules – list of extra modules to load (these are loaded before loading the ‘self’ module)
• verbose – print modules being loaded when trace mode is enabled

make_builddir()

Create the build directory.

make_devel_module(create_in_builddir=False)

Create a develop module file which sets environment based on the build Usage: module load name, which loads the module you want to use. $EBDEVELNAME should then be the full path to the devel module file. So now you can module load $EBDEVELNAME.

WARNING: you cannot unload using $EBDEVELNAME (for now: use module unload basename $EBDEVELNAME)

make_dir(dir_name, clean, dontcreateinstalldir=False)

Create the directory.

make_installdir(dontcreate=None)

Create the installation directory.

make_module_dep(unload_info=None)

Make the dependencies for the module file.

Parameters:unload_info – dictionary with full module names as keys and module name to unload first as corr. value

make_module_deppaths()

Add specific ‘module use’ actions to module file, in order to find dependencies outside the end user’s MODULEPATH.

make_module_description()

Create the module description.

make_module_extend_modpath()

Include prepend-path statements for extending $MODULEPATH.

make_module_extra(altroot=None, altversion=None)

Set extra stuff in module file, e.g. $EBROOT*, $EBVERSION*, etc.

Parameters:

• altroot – path to use to define $EBROOT*
• altversion – version to use to define $EBVERSION*

make_module_extra_extensions()

Sets optional variables for extensions.

make_module_footer()

Insert a footer section in the module file, primarily meant for contextual information

make_module_group_check()

Create the necessary group check.

make_module_req()

Generate the environment-variables to run the module.

make_module_req_guess()

A dictionary of possible directories to look for.

make_module_step(fake=False)

Generate module file

Parameters:fake – generate ‘fake’ module in temporary location, rather than actual module file

mod_subdir

Subdirectory in module install path

moduleGenerator

Module generator (DEPRECATED, use self.module_generator instead).

name

Shortcut the get the module name.

obtain_file(filename, extension=False, urls=None, download_filename=None, force_download=False, git_config=None, no_download=False, download_instructions=None, alt_location=None)

Locate the file with the given name - searches in different subdirectories of source path - supports fetching file from the web if path is specified as an url (i.e. starts with “http://:”) :param filename: filename of source :param extension: indicates whether locations for extension sources should also be considered :param urls: list of source URLs where this file may be available :param download_filename: filename with which the file should be downloaded, and then renamed to <filename> :param force_download: always try to download file, even if it’s already available in source path :param git_config: dictionary to define how to download a git repository :param no_download: do not try to download the file :param download_instructions: instructions to manually add source (used for complex cases) :param alt_location: alternative location to use instead of self.name

package_step()

Package installed software (e.g., into an RPM), if requested, using selected package tool.

patch_step(beginpath=None, patches=None)

Apply the patches

permissions_step()

Finalize installation procedure: adjust permissions as configured, change group ownership (if requested). Installing user must be member of the group that it is changed to.

post_init()

Run post-initialization tasks.

post_install_step()

Do some postprocessing - run post install commands if any were specified

post_iter_step()

Restore options that were iterated over

prepare_for_extensions()

Also do this before (eg to set the template)

prepare_step(start_dir=True, load_tc_deps_modules=True)

Pre-configure step. Set’s up the builddir just before starting configure

Parameters:

• start_dir – guess start directory based on unpacked sources
• load_tc_deps_modules – load modules for toolchain and dependencies in build environment

print_post_install_messages()

Print post-install messages that are specified via the ‘postinstallmsgs’ easyconfig parameter.

remove_module_file()

Remove module file (if it exists), and check for ghost installation directory (and deal with it).

report_test_failure(msg_or_error)

Report a failing test either via an exception or warning depending on ignore-test-failure

Parameters:msg_or_error – failure description (string value or an EasyBuildError instance)

required_linked_shared_libs()

List of shared libraries which must be linked in all installed binaries/libraries. Supported values are pure library names without ‘lib’ prefix or extension (‘example’), file names (‘libexample.so’), and full paths (‘/usr/lib64/libexample.so’).

reset_env()

Reset environment. When iterating over builddependencies, every time we start a new iteration we need to restore the environment to where it was before the relevant modules were loaded.

run_all_steps(run_test_cases)

Build and install this software. run_test_cases (bool): run tests after building (e.g.: make test)

run_post_install_commands(commands=None)

Run post install commands that are specified via ‘postinstallcmds’ easyconfig parameter.

run_step(step, step_methods)

Run step, returns false when execution should be stopped

sanity_check_linked_shared_libs(subdirs=None)

Check whether specific shared libraries are (not) linked into installed binaries/libraries.

sanity_check_load_module(extension=False, extra_modules=None)

Load module to prepare environment for sanity check

sanity_check_rpath(rpath_dirs=None)

Sanity check binaries/libraries w.r.t. RPATH linking.

sanity_check_step(*args, **kwargs)

Do a sanity check on the installation - if any of the files/subdirectories in the installation directory listed

in sanity_check_paths are non-existent (or empty), the sanity check fails

set_parallel()

Set ‘parallel’ easyconfig parameter to determine how many cores can/should be used for parallel builds.

set_up_cuda_cache()

Set up CUDA PTX cache.

short_mod_name

Short module name (not including subdirectory in module install path)

skip_extensions()

Skip already installed extensions, by removing them from list of Extension instances to install (self.ext_instances).

This is done in parallel when EasyBuild is configured to install extensions in parallel.

skip_extensions_parallel(exts_filter)

Skip already installed extensions (checking in parallel), by removing them from list of Extension instances to install (self.ext_instances).

skip_extensions_sequential(exts_filter)

Skip already installed extensions (checking sequentially), by removing them from list of Extension instances to install (self.ext_instances).

skip_step(step, skippable)

Dedice whether or not to skip the specified step.

stage_install_step()

Install in a stage directory before actual installation.

start_dir

Start directory in build directory

test_cases_step()

Run provided test cases.

test_step()

Run unit tests provided by software (if any).

toolchain

Toolchain used to build this easyblock

update_config_template_run_step()

Update the the easyconfig template dictionary with easyconfig.TEMPLATE_NAMES_EASYBLOCK_RUN_STEP names

update_exts_progress_bar(info, progress_size=0, total=None)

Update extensions progress bar with specified info and amount of progress made

version

Shortcut the get the module version.

exception easybuild.framework.easyblock.StopException

Bases: Exception

StopException class definition.

easybuild.framework.easyblock.build_and_install_one(ecdict, init_env)

Build the software :param ecdict: dictionary contaning parsed easyconfig + metadata :param init_env: original environment (used to reset environment)

easybuild.framework.easyblock.build_easyconfigs(easyconfigs, output_dir, test_results)

Build the list of easyconfigs.

easybuild.framework.easyblock.copy_easyblocks_for_reprod(easyblock_instances, reprod_dir)

easybuild.framework.easyblock.get_easyblock_instance(ecdict)

Get an instance for this easyconfig :param ecdict: parsed easyconfig (EasyConfig instance)

returns an instance of EasyBlock (or subclass thereof)

easybuild.framework.easyblock.inject_checksums(ecs, checksum_type)

Inject checksums of given type in specified easyconfig files

Parameters:

• ecs – list of EasyConfig instances to inject checksums into corresponding files
• checksum_type – type of checksum to use

easybuild.framework.easyblock.inject_checksums_to_json(ecs, checksum_type)

Inject checksums of given type in corresponding json files

Parameters:

• ecs – list of EasyConfig instances to calculate checksums and inject them into checksums.json
• checksum_type – type of checksum to use

easybuild.framework.easyblock.print_dry_run_note(loc, silent=True)

Print note on interpreting dry run output.

easybuild.framework.easyblock.reproduce_build(app, reprod_dir_root)

Create reproducibility files (processed easyconfig and easyblocks used) from class instance

Parameters:

• app – easyblock class instance
• reprod_dir_root – root directory in which to create the ‘reprod’ directory

Returns:

reprod_dir directory containing reproducibility files