-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
Description
According to the NERSC HPSS documentation (https://docs.nersc.gov/filesystems/archive/#avoid-very-large-files), files over 2TB are inefficient when put to HPSS. They recommend breaking files up into 500GB chunks if they get over that limit.
The hsi handler mcscript.task.archive_handler_hsi() should:
- Inspect the size of an archive file to put.
- Put the file directly using
hsiif smaller than threshold, or - use
splitto break up the file into smaller segments and put those withhsi.
In addition (so that consumers of archives don't need to be aware of this splitting behavior):
-
mcscriptshould also provide a wrapper function to fetch and reassemble archives,
Reactions are currently unavailable