This guide will help you get started with using the EasyBuild repository for local software installations on the Michigan State University (MSU) High Performace Computing Center (HPCC). Follow the steps below to set up your environment, build software, and manage your installations.
- First Time Setup Guide
- Advanced Topics
- Hello World Example
- Managing Your EasyBuild Local
- Common Problems
- Additional Resources
If this is your first time using EasyBuild or havn't used it in a while, I suggest you take a look at the EasyBuild and Module System Terminology and Commands to get a better understanding of the terms used in this guide and what is happening behind the scenes when running these commands. If you feel confident in your understanding of EasyBuild, you can skip to the Hello World Example to make sure everything is working correctly.
First, clone the repository to your home directory on the HPCC:
git clone https://github.com/colbrydi/easybuild_local.git
Navigate to the repository directory:
cd easybuild_local
To begin using EasyBuild in this repository, you need to set up your environment by sourcing the activate_easybuild_local.sh script. This script loads the necessary modules and points EasyBuild to the local configuration file.
Run the following command:
source activate_easybuild_local.sh
This will load EasyBuild and configure it to use the paths defined in config.cfg.
NOTE: This tutorial teaches how to download and install files in the easyconfig directory. Later on there are instructions on how to download and install files from the HPCC system repository.
Once your environment is set up, you can start installing software locally using EasyBuild. The repository contains EasyBuild configuration files (.eb files) for several software packages in the easyconfigs/ directory.
To install a software package, run:
eb --parallel=8 --robot ./easyconfigs/<FILENAME>.eb
The parallel command determines how many cores are being used and robot automatically downloads all of the dependencies in an EasyBuild config. Replace <FILENAME> with the name of the configuration file you want to build.
For example, you can run the example eb file provided by typing the following:
eb --parallel=8 --robot ./easyconfigs/Hello-2.10-GCCcore-11.2.0.eb
After the installation is complete, you can make the locally installed software available by running the following script:
source eb_local_use.sh
This adds the locally installed modules to the module search path, allowing you to load and use them.
For example, to load the provided example, you would run:
module load Hello/2.10-GCCcore-11.2.0
Now that the module is loaded, you can use the software as needed. For this example, you can run:
hello
And you should see the following output:
Hello, world!
This repository includes a simple "Hello, World!" example to demonstrate the process of building and installing software using EasyBuild. These are similar to the steps above.
git clone https://github.com/colbrydi/easybuild_local.git
cd easybuild_local
source activate_easybuild_local.sh
eb --parallel=8 --robot ./easyconfigs/Hello-2.10-GCCcore-11.2.0.eb
source eb_local_use.sh
module load Hello/2.10-GCCcore-11.2.0
hello
This prints Hello, world!
In order to find dependencies that are required to download and find software on the HPCC, there is an EasyBuild command that is helpful
eb -S <SOFTWARE>
This command searches for EasyBuild files with the name of that are available to download.
Once you have found the EasyBuild file you want to download, run:
eb -M <FILENAME>.eb
This command shows all the dependencies that are needed to download to run the EasyBuild file.
To download the dependencies, run:
eb --parallel=8 <FILENAME>.eb
This command downloads the dependency. When downloading the dependencies, make sure to download the dependency that has the same .eb as the eb -M search last. In this case, I would download the dependency RNA-SeQC-1.1.8-GCCcore-11.2.0-Java-11.eb last.
Dowloading all the necessary dependencies can be very tedious. In order to download all the dependencies and the EasyBuild file at once, use
eb --parallel=8 --robot <FILENAME>.eb
This allows all the dependencies to be downloaded at once rather than having to download them individually.
This guide explains how to install software from a specific commit in the EasyBuild repository. For this example, we will use OpenFOAM v2112.
- Find the software you want to install in the easyconfigs folder: https://github.com/easybuilders/easybuild-easyconfigs/tree/develop/easybuild/easyconfigs
- Example for OpenFOAM v2112: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/o/OpenFOAM/OpenFOAM-v2112-foss-2023a.eb
- Click on History to view previous commits to the file.
- Identify the most recent commit that you want to install from.
- Copy the full commit SHA (a long alphanumeric string).
Use the following command to download the software from the commit:
eb --from-commit <commit-SHA> <easyconfig-file>For OpenFOAM v2112, the command looks like this:
eb --from-commit 70e5f79258b1cb9a429525b774e4131486210e18 OpenFOAM-v2112-foss-2023a.ebNote: Installations may take a while so it is recommended to perform this on an Interactive Desktop.
This guide explains how to install software from a pull request in the EasyBuild repository. For this example, we will use OpenFOAM v2112.
Go to https://github.com/easybuilders/easybuild-easyconfigs and click on the easybuild/easyconfigs folder.
Scroll through and click on the easybuild configuration you want to download. In this example we want OpenFOAM-v2112-foss-2023a.eb 
Once you find the file you want to download, click on the history button in the top right corner.
Once on the commits screen, click on the commit you want to download. In this case, the “fix dependency” version is being downloaded.
The pull number will be found in the top left corner next to the blue “develop”. The pull number for this OpenFOAM easyconfig is 22241.
To download OpenFOAM from the pull request number, run:
eb --from-pr [pull request number] --robot
For this version of OpenFOAM the command is:
eb --from-pr 22241 --robot.
Note: It is recommended to download the EasyConfig file on the HPCC Interactive Desktop because some downloads may take hours.
ssh dev-intel16
Navigate to the repository directory:
cd easybuild_local
Run the following command:
source activate_easybuild_local.sh
This will load EasyBuild and configure it to use the paths defined in config.cfg.
To download the R 4.4.1 GitHub file, run:
wget https://raw.githubusercontent.com/easybuilders/easybuild-easyconfigs/refs/heads/develop/easybuild/easyconfigs/r/R/R-4.4.1-gfbf-2023b.eb
(URL is from https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/r/R/R-4.4.1-gfbf-2023b.eb) Replace the URL with the appropriate R GitHub file if needed.
Move the download to the easyconfigs folder:
mv R/4.4.1-gfbf-2023b.eb easyconfigs
Open the file:
vi R-4.4.1-gfbf-2023b.eb
Make the following edits to include intel-16:
version = '4.4.1-intel16'
toolchain = {'name': 'intel', 'version': '16'}
versionsuffix
Save a copy of the file to include intel16:
cp R-4.4.1-gfbf-2023b.eb R-4.4.1-gfbf-2023b-intel16.eb
Run the following command to install R-4.4.1 using the new .eb file on dev-intel16:
eb --parallel=8 --robot ./easyconfigs/R-4.4.1-gfbf-2023b-intel16.eb
Check that a new module with intel16 exists:
ls ./software/modules/all
Navigate to the R folder:
cd ./software/modules/all/R
After the installation is complete, you can make the locally installed software available by running the following script:
module use ~/easybuild_local/software/modules/all
To load the new module, run the following script:
module load R/4.4.1-gfbf-2023bintel16.lua
Now that the module is loaded, you can use the software as needed. For this example, you can run:
R
And you should be able to use R as needed.
Double check that the working version of R points a install path with intel16 in the name:
which R
You should see something like:
/mnt/ufs18/home-067/nguye922/easybuild_local/software/software/R/4.4.1-gfbf-2023bintel16/bin/R
- Repeat steps from dev-intel16, but change intel-16 to intel-18.
- At Step 6, make sure that it is pointing to a different R install in "software" than intel16.
If you need to clean up your environment and reset the installation directories, you can use the eb_local_reset.sh script. This script will delete the software/ directory and recreate the necessary directories for a fresh start. This will also deactivate your conda environment.
To reset the environment, run:
source eb_local_reset.sh
If you run into an error like this:
FAILED: Installation ended unsuccessfully
(build directory: /mnt/ufs18/home-251/wunderky/easybuild_local/tempdir/build/Hello/2.10/GCCcore-11.2.0):
build failed (first 300 chars):
Module is already loaded (EBROOTHELLO is set), installation cannot continue. (took 3 secs)
Here are some steps to try and resolve the issue:
- Unload the module before running the installation command:
module unload Hello
- See if the environment variable is set:
env | grep EBROOTHELLO
If the environment variable is set, unset it:
unset EBROOTHELLO
- Try resetting the environment and starting fresh:
Older tool chains may cause errors while downloading. Some EB files as early as 2021b have been successfully downloaded, but earlier toolchains could cause traceback errors.
Toolchains that are old or cause errors can be updated. There are two commands that can update EasyBuild files in order to be downloaded.
--experimental --try-update-deps
This command updates the dependencies in order to successfully download the file if it previously failed.
--try-toolchain-version <VERSION>
This command specifically updates the toolchain to see if download will be successful on a different version. In the example below, the earlier toolchain version GCCcore-12.3.0 was used after the download on GCCcore-11.3.0 failed.
Nick Panchy
Connor Neiheisel
Anna Nguyen