Update resource requirements for LoTSS subtraction.#87
Update resource requirements for LoTSS subtraction.#87
Conversation
steps/box_predict.cwl
Outdated
| - entry: $(inputs.ms) | ||
| writable: true | ||
| - entry: $(inputs.solsdir) | ||
| writable: true |
There was a problem hiding this comment.
Why does this needs to be writable? Given that we ran this successfully without making it writable?
There was a problem hiding this comment.
I and Gabriella couldn't run it successfully without that folder being writable recently for some reason. I'm not sure why.
There was a problem hiding this comment.
It comes at a cost of an extra (maybe unnecessary) copy. Maybe worthwhile to figure out why this is suddenly needed.
There was a problem hiding this comment.
New run is running that has the MSes in SOLSDIER renamed to the pipeline outputs instead of symlinked to the ddf outputs. Let's see if that works better.
There was a problem hiding this comment.
Renaming the ddf-pipeline's output instead of adding a symlink to them has worked, so for some reason it doesn't like the symlink. We can just rename them before running the workflow, so I have removed the writable flag for SOLSDIR.
tikk3r
force-pushed
the
slurm-resources-subtraction
branch
from
f49ec4b to
cd2662b
Compare
tikk3r
force-pushed
the
slurm-resources-subtraction
branch
from
cd2662b to
33ac785
Compare
3 participants
This PR adds resource requirements based on running the subtraction on a Lockman pointing on cosma.
DIS2 h5parm merging
Subtracting one chunk
Looking at the output of
timeit seems the prediction only uses ~45 GBthe the OoM killer kept kicking in and DDFacet itself reported ~125GB so perhaps there's a small spike somewhere or something like that. That is where the 128 GB ramMin comes from.