Skip to content

Use SLURM singleton dependency for GPU jobs to prevent conflicts #530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JRPan merged 1 commit into from
Jan 28, 2026
+8 −18
Merged

Use SLURM singleton dependency for GPU jobs to prevent conflicts #530

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 3 additions & 13 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ jobs:
./util/job_launching/monitor_func_test.py -v --sleep_time 300 -s stats-per-app-ptx.csv -N short-ptx-${{ github.run_number }}_${{ github.run_attempt}}
Tracer-Tool:
needs: [check-format]
runs-on: tgrogers-gpu01
runs-on: tgrogers-raid
timeout-minutes: 720
defaults:
run:
Expand Down Expand Up @@ -295,16 +295,6 @@ jobs:
source ./gpu-simulator/setup_environment.sh
make clean -C gpu-simulator
make -j20 -C gpu-simulator
- name: test-prebuilt-traces
run: |
source ./env-setup/12.8_env_setup.sh
source ./gpu-simulator/setup_environment.sh
./get-accel-sim-traces.py -a tesla-v100/rodinia_2.0-ft
cd hw_run; tar -xzvf rodinia_2.0-ft.tgz; cd -
./util/job_launching/run_simulations.py -B rodinia_2.0-ft -C QV100-SASS -T ./hw_run/ -N rodinia_2.0-ft-online-$$
./util/job_launching/monitor_func_test.py --sleep_time 300 -N rodinia_2.0-ft-online-$$ -v
rm -rf hw_run
rm -rf sim_run_11.0
- name: Build Tracer
run: |
source ./env-setup/12.8_env_setup.sh
Expand All @@ -324,12 +314,12 @@ jobs:
source ./env-setup/12.8_env_setup.sh
source ./gpu-app-collection/src/setup_environment
rm -rf ./hw_run/
./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft -D 7
srun --job-name=gpu-lock --dependency=singleton --partition=tgrogers-dgx -- ./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft -D 7
- name: generate-rodinia_2.0-ft-hw_stats
run: |
source ./env-setup/12.8_env_setup.sh
source ./gpu-app-collection/src/setup_environment
./util/hw_stats/run_hw.py -B rodinia_2.0-ft -D 7
srun --job-name=gpu-lock --dependency=singleton --partition=tgrogers-dgx -- ./util/hw_stats/run_hw.py -B rodinia_2.0-ft -D 7
- name: test-new-traces
run: |
source ./env-setup/12.8_env_setup.sh
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/weekly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
Tracer-Weekly:
timeout-minutes: 720
if: ${{ github.repository == 'accel-sim/accel-sim-framework' || github.event_name == 'workflow_dispatch' }}
runs-on: tgrogers-gpu01
runs-on: tgrogers-raid
defaults:
run:
shell: bash
Expand Down Expand Up @@ -52,8 +52,8 @@ jobs:
source ./env-setup/12.8_env_setup.sh
source ./gpu-app-collection/src/setup_environment
rm -rf ./hw_run/
./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft,rodinia-3.1,GPU_Microbenchmark -D 7
./util/hw_stats/run_hw.py -B rodinia_2.0-ft,rodinia-3.1,GPU_Microbenchmark -D 7
srun --job-name=gpu-lock --dependency=singleton --partition=tgrogers-dgx -- ./util/tracer_nvbit/run_hw_trace.py -B rodinia_2.0-ft,rodinia-3.1,GPU_Microbenchmark -D 7
srun --job-name=gpu-lock --dependency=singleton --partition=tgrogers-dgx -- ./util/hw_stats/run_hw.py -B rodinia_2.0-ft,rodinia-3.1,GPU_Microbenchmark -D 7
rm -rf /scratch/tgrogers-disk01/a/common/for-sharing/$USER/nightly-traces
mkdir -p /scratch/tgrogers-disk01/a/common/for-sharing/$USER/nightly-traces
mv ./hw_run /scratch/tgrogers-disk01/a/common/for-sharing/$USER/nightly-traces/hw_run
Expand All @@ -63,9 +63,9 @@ jobs:
source ./env-setup/12.8_env_setup.sh
source ./gpu-app-collection/src/setup_environment
rm -rf ./hw_run/
./util/tracer_nvbit/run_hw_trace.py -B Spinlock -D 7 --spinlock_handling fast_forward
srun --job-name=gpu-lock --dependency=singleton --partition=tgrogers-dgx -- ./util/tracer_nvbit/run_hw_trace.py -B Spinlock -D 7 --spinlock_handling fast_forward
mv ./hw_run /scratch/tgrogers-disk01/a/common/for-sharing/$USER/nightly-traces/hw_run_fast_forward
./util/tracer_nvbit/run_hw_trace.py -B Spinlock -D 7 --spinlock_handling none
srun --job-name=gpu-lock --dependency=singleton --partition=tgrogers-dgx -- ./util/tracer_nvbit/run_hw_trace.py -B Spinlock -D 7 --spinlock_handling none
mv ./hw_run /scratch/tgrogers-disk01/a/common/for-sharing/$USER/nightly-traces/hw_run_none
SASS-Weekly:
timeout-minutes: 720
Expand Down
Loading