-
Notifications
You must be signed in to change notification settings - Fork 119
Description
Description
Currently users can start up a sandbox like nvidia/cuda:12.9.1-devel-ubuntu24.04 to access the host's NVIDIA GPU drivers and run CUDA applications with Enroot, as stated in the Usage documentation.
But sometimes there is a need to run an image built without NVIDIA / CUDA support, for example debian:bookworm-slim. It would be great if Enroot could provide access to the host's NVIDIA GPU drivers in this case as well, similar to how Docker and Podman do it.
Environment
- Host OS: Rocky Linux 8.10
- Enroot version: 3.5.0
- NVIDIA driver version: 570.153.02
Expected Behavior
The nvidia-smi command works inside an Enroot sandbox started from an image without NVIDIA / CUDA support, as long as the host has the NVIDIA drivers installed and running, like in Docker or Podman:
podman run --rm --device nvidia.com/gpu=all debian:bookworm-slim nvidia-smiOutput example:
Wed Dec 17 07:18:11 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.153.02 Driver Version: 570.153.02 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 D Off | 00000000:41:00.0 Off | Off |
| 31% 26C P8 15W / 425W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Current Behavior
Currently, if I start an Enroot sandbox from an image without NVIDIA / CUDA support, no NVIDIA libraries or binaries are available inside the container, so running nvidia-smi fails:
enroot start --root --rw debian+bookworm-slim nvidia-smiOutput example:
/etc/rc: line 9: exec: nvidia-smi: not found
Workaround
My workaround is to manually mount the host's nvidia-smi binary and the relevant libnvidia-ml.so.1 libcuda.so.1 libraries into the container.
The NVIDIA driver from my host was installed via a .run file. The relevant files are located in /usr/lib64/ and /usr/bin/. Here is the command:
enroot start \
--root \
--rw \
--env LD_LIBRARY_PATH=/usr/lib64 \
--mount /usr/lib64/libcuda.so.1:/usr/lib64/libcuda.so.1 \
--mount /usr/lib64/libnvidia-ml.so.1:/usr/lib64/libnvidia-ml.so.1 \
--mount /usr/bin/nvidia-smi:/usr/bin/nvidia-smi \
debian+bookworm-slim \
nvidia-smiOutput example:
Wed Dec 17 15:40:17 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.153.02 Driver Version: 570.153.02 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 D Off | 00000000:41:00.0 Off | Off |
| 30% 26C P8 15W / 425W | 1MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Applications like PyTorch(with CUDA runtime provided through PyPI packages) can now access the GPU as expected.
Additional Information
Each time I start an Enroot sandbox, there is an error message popping up:
/etc/enroot/hooks.d/10-shadow.sh: line 70: 312531 Broken pipe yes 2> /dev/null
312532 Segmentation fault (core dumped) | pwck -R "${ENROOT_ROOTFS}" "${pwddb#${ENROOT_ROOTFS}}" /etc/shadow > /dev/null 2>&1
This seems unrelated to the NVIDIA driver, but I thought I should mention it in case it is relevant.