From 8ead1b8ebd4250829eb36c065d01c0a176c42de6 Mon Sep 17 00:00:00 2001 From: Alex Wadell Date: Wed, 22 Jan 2025 11:40:05 -0500 Subject: [PATCH 1/2] Julia precompile recs. and add uArch to hardware --- about/hardware.md | 8 +++---- about/user_software.md | 51 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 4 deletions(-) diff --git a/about/hardware.md b/about/hardware.md index 3c91f1a..ced3b17 100644 --- a/about/hardware.md +++ b/about/hardware.md @@ -26,10 +26,10 @@ math: mathjax2 | Largemem | 3 | AMD 9654 | | 768 GB | 1.9TB | 13,989 | | CPU | 25 | AMD 9654 | | 368 GB | 1.9TB | 12,998 | -| CPU | Cores | Threads | Base | Boost | L3 Cache | -| ---------------------------------------------------------------------- | ----- | ------- | ------ | ------------------ | -------- | -| [AMD Epyc 9654 CPU](https://www.amd.com/en/products/cpu/amd-epyc-9654) | 96 | 192 | 2.6GHz | 3.55GHz (All Core) | 384MB | -| [AMD Epyc 7513 CPU](https://www.amd.com/en/products/cpu/amd-epyc-7513) | 32 | 64 | 2.6GHz | 3.65GHz (Max) | 128MB | +| CPU | μArch | Cores | Threads | Base | Boost | L3 Cache | +| ---------------------------------------------------------------------------------------------------------------------------------- | ----- | ----- | ------- | ------ | ------------------ | -------- | +| [AMD Epyc 9654 CPU](https://www.amd.com/en/products/processors/server/epyc/4th-generation-9004-and-8004-series/amd-epyc-9654.html) | Zen4 | 96 | 192 | 2.6GHz | 3.55GHz (All Core) | 384MB | +| [AMD Epyc 7513 CPU](https://www.amd.com/en/products/processors/server/epyc/7003-series/amd-epyc-7513.html) | Zen3 | 32 | 64 | 2.6GHz | 3.65GHz (Max) | 128MB | > Nodes are partitioned by *threads*, not *cores*. Picking 1 or a multiple of 2 is advisable; see [sbatch's](https://slurm.schedmd.com/sbatch.html) `--distribution` flag diff --git a/about/user_software.md b/about/user_software.md index 2eb9921..b528c5f 100644 --- a/about/user_software.md +++ b/about/user_software.md @@ -147,3 +147,54 @@ packages: scalapack: [amdsclapack, netlib-scalapack] ``` + +## Julia + +## Minimizing Redundant Precompilation + +As Artemis is a [heterogeneous cluster](./hardware.md#cluster-architecture), switching between different partitions can trigger frequent precompilation, as Julia attempts to recompile for the current microarchitecture. To avoid this set [`JULIA_CPU_TARGET`](https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_CPU_TARGET) to an appropriate setting. + +For example, within your `~/.bashrc` or equivalent, put: + +```shell +export JULIA_CPU_TARGET="generic;znver4,clone_all;znver3,clone_all;haswell" +``` + +This will instruct Julia to compile for Zen4 (Most Worker Nodes), Zen3 (A100 Nodes) and haswell (Head node), with a generic CPU target as a fallback. This will increase the size of `~/.julia/compiled` as 3 sets of system and package images will now need to be stored. For more information see the [Julia docs](https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_CPU_TARGET). + + +### Example Precompile Job Script +```shell +#!/bin/bash +#SBATCH -p venkvis-cpu +#SBATCH --cpus-per-task 4 +#SBATCH --mem-per-cpu 1800M +#SBATCH --time=0:20:00 +set -x +my_job_header + +# Configure Julia +export JULIA_CPU_TARGET="generic;znver4,clone_all;znver3,clone_all;haswell" +export JULIA_PKG_PRECOMPILE_AUTO=0 # Disable automatic precompilation so we control when it happens (Mainly for MPI) +export JULIA_NUM_PRECOMPILE_TASKS=$SLURM_CPUS_ON_NODE # Limit precompilation tasks to the number of CPUs + +# Uncomment if using MPI.jl +# module --ignore_cache load openmpi/4.1.6 +# julia --color=no --startup-file=no --project -e 'using MPIPreferences; MPIPreferences.use_system_binary()' + +# Precompile the Project +julia --color=no --startup-file=no --project -e 'using Pkg; Pkg.resolve(); Pkg.instantiate(); Pkg.precompile(timing=true)' +``` + +The job script would then look like: + +```shell +# [Job Header Goes Here] + +export JULIA_CPU_TARGET="generic;znver4,clone_all;znver3,clone_all;haswell" + +julia --startup-file=no --project ... +``` + +> Note: `--startup-file=no` is needed to avoid loading packages from your base environment (i.e. Revise) and + From a85e08e4bc2a20e044267a1ae93390d5303c898d Mon Sep 17 00:00:00 2001 From: Alex Wadell Date: Wed, 22 Jan 2025 20:16:44 +0000 Subject: [PATCH 2/2] fix formatting, add plug for relative links --- Gemfile | 2 ++ _config.yml | 3 +++ about/user_software.md | 1 - 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/Gemfile b/Gemfile index 465d6d3..3359078 100644 --- a/Gemfile +++ b/Gemfile @@ -5,3 +5,5 @@ gem "jekyll", "~> 4.3.3" gem "html-proofer", "~> 5.0" gem "just-the-docs", "~> 0.8.2" + +gem "jekyll-relative-links", "~> 0.7.0" diff --git a/_config.yml b/_config.yml index 9e3d585..812310a 100644 --- a/_config.yml +++ b/_config.yml @@ -13,3 +13,6 @@ ga_tracking: G-X2LPWRLJ7X includes: head: - head_custom.html + +plugins: + - jekyll-relative-links diff --git a/about/user_software.md b/about/user_software.md index b528c5f..bbb7512 100644 --- a/about/user_software.md +++ b/about/user_software.md @@ -197,4 +197,3 @@ julia --startup-file=no --project ... ``` > Note: `--startup-file=no` is needed to avoid loading packages from your base environment (i.e. Revise) and -