Add controls for attaching nodes/task info to jobspecs and launchers #460
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds additional control on how resource specs are attached to both batch jobs and launchers separately. Enables use of more dynamic resource configuration such as the gpumode introduced with flux for modern AMD machines which can change the number of logical gpus after job scheduling time. Pre 1.1.12 behavior attaches tasks to the jobspec in flux, which can result in 'unsatisfiable' job errors due to there not being sufficient logical gpu's at jobspec validation time to fulfill the job request. This case requires not binding the tasks to the jobspec, but still binding the tasks to the $(LAUNCHER) generated flux run.
This initial version adds support for flux only; slurm/lsf/etc handling to follow in subsequent release (will be no-op everywhere but flux scheduled jobs).