Latest CycleCloud (8.8.1) and Slurm clusters seems to have initialisation issues with GPU Nodes?

[garymansellricardo]

I have just installed CycleCloud 8.8.1, Slurm Template 4.0.4 and running Slurm 25.05.2 which comes with it all.

Normal nodes seem to be fine, but GPU Enabled nodes all throw up initialisation errors in the CC UI and show as failed (and pending/drained with sinfo) before they get terminated and another node is tried repeatedly. I have tried this with OOTB Alma 8, Alma 9 and my own Custom Alma 9 images for both scheduler and execute nodes.

I have also seen these errors on another GPU node:

ChatGPT seems to suggest that at least some of these are benign and that it is just the newer version of CC or the software versions in the latest HPC images that is being overly "picky" testing stuff, but even so, it still stops my GPU solving jobs from running.

Any ideas what is causing this and how to remedy?

[aditigaur4]

For the persistent mode,

On the gpu nodearray you can set the cloud-init nvidia-smi -pm 1 to set persistent mode. That will make that error go away. This error comes from DCGM health checks. This error does not come in the newer nvidia sku's because nvidia changed how they set the persistent mode. But in some older skus this needs to be run.

As far as the nvlink one, that one is important. So may I ask which sku you are using?

[garymansellricardo]

Thanks so much @aditigaur4 - adding the "nvidia-smi -pm 1" command to my cloud-init for the GPU nodearrays has fixed the issue and they initialise correctly without error again now and I can run my analysis jobs again!

FYI - I am running on NC24adsA100_v4 and NC40adsH100_v5 nodes, so perhaps older skus.

I can't seem to reproduce the nvlink error I posted again (have tried in 3x regions and both skus in each), I think I only saw it once - maybe there was a hardware issue, so I will ignore for now.

[garymansellricardo]

Sorry, me again....

I have just seen the NVLink Errors again in my cluster in GWC region on NC40adsH100v5 nodearray (the old NC24adsA100v4 nodearray came up fine, though and it uses the same node image):

I left it for half an hour and then tried again and this time the NC40adsH100v5 nodearray came up OK.

I then tried starting three nodes in the array and two came up and one exhibited the error:

What do you think - is this a dodgy bit of NVLink hardware in one/some of the GWC Region's H100 VM's do you think, or is there still a software/image issue here?

[aditigaur4]

Hello

So a bit of background-- healthagent is running on every node as part of cyclecloud node health checks and more information about the project is available here in the Readme: https://github.com/Azure/cyclecloud-healthagent

For GPU's, healthagent relies on DCGM to detect health issues. So in this case we are relaying the information from DCGM to both cyclecloud UI and to slurm (through slurm healthcheck program thats configured in slurm.conf) which drains the node.

so you can verify the nvlink issue through DCGM as well -- dcgmi nvlink -s which should show the same thing you are seeing.

Some virtualized skus based on h100 hardware, we are seeing the nvlinks show up as down (and that is expected...an h100 system contains 8 gpus and18 nvlinks). But skus virtualized on top of it contain less so they may contain 12 nvlinks for example. This shows up in DCGM as nvlinks down and dcgm flags it as an error but for some skus this may be expected.

If you are workload is not senstive to nvlinks and/or this is the only issue healthagent is catching-- you can disable healthcheckprogram in slurm.conf by setting healthcheckinterval to 0.

Note this will not stop healthagent from running as its a persistent systemd service and it will still continue showing you errors in the CC dashboard but it will prevent slurm from draining these nodes.

We are working on a patch that would fix this problem for these skus.

Note: This should only be a problem I believe on skus with more than 1 gpu where some nvlinks are expected to be down. IF you only have single GPU, I think healthagent will not flag the node.

let me know if this helps.