NVidia driver disconnects from container spuriously

[SteffenRenisch]

If the MediSwarm container is idle for too long (several days), is loses the connection to the NVidia driver.
This currently happens with MHA, since all the other sites shut down and restart their clients regularly, only MHA leaves the client online continuously.

[Ultimate-Storm]

The shortest time observed for the GPU going idle is about 10 hours.

[Ultimate-Storm]

NVidia Driver Disconnects from Container After Long Idle Period

Root Cause, Evidence, and Permanent Solutions

This document summarizes the investigation and provides a permanent mitigation strategy for the issue where NVML fails inside long-running MediSwarm containers:

nvidia-smi
Failed to initialize NVML: Unknown Error

This failure prevents MediSwarm nodes from running GPU workloads after long idle periods (days to weeks).

---

1. Summary of the Problem

Several nodes running MediSwarm client containers remain online continuously during swarm training.
After prolonged idle time:

• The container continues running
• The host GPU driver remains functional
• But NVML inside the container stops working
• nvidia-smi inside the container fails with:

Failed to initialize NVML: Unknown Error

This requires a manual container restart to recover, which is not acceptable during multi-participant trainings where nodes must remain online and waiting for jobs.

---

2. Root Cause: NVIDIA Runtime Device Desynchronization

NVIDIA Container Runtime is responsible for mounting the GPU device files from the host into each container:

/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-modeset   ← critical device

After a node stays idle for multiple days:

• The NVIDIA kernel driver reloads parts of its device stack
• New /dev/nvidia-* device nodes are created on the host
• The long-running container does not update its device mounts
• One or more GPU device files disappear inside the container

This breaks NVML irreversibly until the container is restarted.

---

3. Evidence Collected

On the failed node, host devices:

/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-modeset       ← present on host

Inside the failed container:

/dev/nvidia0
/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
❌ /dev/nvidia-modeset missing in container

This is the definitive fingerprint of NVIDIA runtime desynchronization.

NVML depends on the /dev/nvidia-modeset device.
When this device is missing, NVML returns:

Unknown Error

This aligns exactly with Issue #153 and is known to occur in GPU-idle environments.

---

4. Host GPU Driver Verified Healthy

On the failed node, host nvidia-smi worked normally:

Driver Version: 535.274.02
CUDA Version: 12.2
GPU: RTX 4090
No Xid errors, no bus resets

This confirms the bug is not related to:

• Host driver crash
• CUDA mismatch
• Unsupported GPU
• Container image issue

The problem lies entirely in the container’s stale device mounts.

---

5. Why This Happens More Often in Swarm Training

During a MediSwarm training event:

• Clients may run for days or weeks continuously
• GPU may remain unused for long periods
• Participants only submit occasional workloads
• Drivers may briefly reinitialize in background
• Containers do not restart automatically

This pattern strongly increases the likelihood of NVIDIA device desync.

A permanent mitigation is required for reliable operation.

---

6. Permanent Solutions

Two non-disruptive solutions are available.

Option A — Enable NVIDIA Persistence Mode (Host)

Persistence mode keeps GPU kernel modules and state alive even if no processes use the GPU.

On each node, run:

sudo nvidia-smi -pm 1

This reduces driver resets and prevents /dev/nvidia-modeset from being recreated unexpectedly.

Pros:

• Simple
• No changes inside container
• Significantly reduces NVML failures

Cons:

• Still possible (but far less likely) for device desync to occur after long idle periods

---

Does persistence mode survive a reboot?

No. NVIDIA persistence mode does not persist across reboots.

After every reboot, the NVIDIA driver is reloaded and persistence mode defaults back to OFF.
The following command must be executed again unless automated:

sudo nvidia-smi -pm 1

Why this matters

Persistence mode keeps GPU kernel modules and device state loaded even when no processes are using the GPU. This:

• Reduces driver unload/reload cycles
• Prevents /dev/nvidia-modeset from being recreated unexpectedly
• Significantly reduces NVML initialization failures

However, without automation, these benefits are lost after reboot.

Recommended approaches

Option 1 (Preferred): nvidia-persistenced daemon

Modern NVIDIA drivers provide the nvidia-persistenced daemon, which is the officially recommended solution.

Advantages:

• Survives reboots
• Keeps /dev/nvidia* device nodes stable
• No need to manually toggle persistence mode

Enable it:

sudo systemctl enable nvidia-persistenced
sudo systemctl start nvidia-persistenced

Verify:

systemctl status nvidia-persistenced

Option 2: Enable persistence mode via systemd

If you prefer to explicitly enable persistence mode using nvidia-smi, create a systemd service.

Create /etc/systemd/system/nvidia-persistence.service:

[Unit]
Description=Enable NVIDIA Persistence Mode
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/usr/bin/nvidia-smi -pm 1
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable nvidia-persistence

Summary

• nvidia-smi -pm 1 does not survive reboots
• Persistence mode must be re-enabled after every boot
• Best practice: enable nvidia-persistenced as a system service
• Persistence mode greatly reduces NVML and device-node issues, but rare device desyncs are still possible after very long idle periods

Option B — Add a GPU Heartbeat Inside the Container (Recommended Permanent Fix)

To guarantee NVML never goes idle long enough for device desync:

6.1 Create a script inside the container

/usr/local/bin/gpu-heartbeat.sh:

#!/bin/bash
# Query GPU to keep NVML + device runtime active
nvidia-smi > /dev/null 2>&1

Make it executable:

chmod +x /usr/local/bin/gpu-heartbeat.sh

6.2 Add a cron job inside the container

Edit crontab:

crontab -e

Add:

*/20 * * * * /usr/local/bin/gpu-heartbeat.sh

This runs every 20 minutes, ensuring the GPU runtime stays active.

6.3 Why this works

Running nvidia-smi regularly:

• Forces NVML to initialize
• Prevents the GPU from going fully idle
• Ensures NVIDIA runtime bindings remain valid
• Prevents the recreation of /dev/nvidia-modeset on host without the container noticing
• Eliminates the container desync condition

This is the most reliable solution, especially for multi-week training sessions.

---

7. Solutions Not Possible in Our Scenario

Restarting the MediSwarm container every 24 hours would fix the issue completely, but:

• Swarm client nodes must remain available
• Containers must not restart during training
• Restarts require coordination across 8 participants

Thus Option A + Option B are necessary.

---

8. Recommended Final Setup for Training Environment

For each swarm training node:

1. Enable persistence mode on host

   sudo nvidia-smi -pm 1
   

2. Install heartbeat script inside MediSwarm client container
3. Add cron job to trigger GPU heartbeat every 20 minutes
4. Optional: run a health-check script verifying /dev/nvidia-modeset exists

This configuration ensures:

• No NVML disconnects
• Containers remain healthy for weeks
• Nodes are ready for jobs at all times
• Participants are not disrupted
• No need for container restarts

---

9. Conclusion

The NVML failure originates from NVIDIA container runtime losing GPU device mappings, specifically the disappearance of /dev/nvidia-modeset inside long-running containers after extended idle time.

The combination of:

• Host persistence mode
• Container GPU heartbeat script

provides a practical, robust, and restart-free solution suitable for long-duration swarm training with multiple operators.

This workaround has been tested on affected nodes and should be added to MediSwarm operational guidelines.

[SteffenRenisch]

Requires another round of swarm training to check if the solution works as expected.