Skip to content

Add NVIDIA / CUDA driver support for images built without it #258

@NewComer00

Description

@NewComer00

Description

Currently users can start up a sandbox like nvidia/cuda:12.9.1-devel-ubuntu24.04 to access the host's NVIDIA GPU drivers and run CUDA applications with Enroot, as stated in the Usage documentation.

But sometimes there is a need to run an image built without NVIDIA / CUDA support, for example debian:bookworm-slim. It would be great if Enroot could provide access to the host's NVIDIA GPU drivers in this case as well, similar to how Docker and Podman do it.

Environment

  • Host OS: Rocky Linux 8.10
  • Enroot version: 3.5.0
  • NVIDIA driver version: 570.153.02

Expected Behavior

The nvidia-smi command works inside an Enroot sandbox started from an image without NVIDIA / CUDA support, as long as the host has the NVIDIA drivers installed and running, like in Docker or Podman:

podman run --rm --device nvidia.com/gpu=all debian:bookworm-slim nvidia-smi

Output example:

Wed Dec 17 07:18:11 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.153.02             Driver Version: 570.153.02     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 D      Off |   00000000:41:00.0 Off |                  Off |
| 31%   26C    P8             15W /  425W |       1MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Current Behavior

Currently, if I start an Enroot sandbox from an image without NVIDIA / CUDA support, no NVIDIA libraries or binaries are available inside the container, so running nvidia-smi fails:

enroot start --root --rw debian+bookworm-slim nvidia-smi

Output example:

/etc/rc: line 9: exec: nvidia-smi: not found

Workaround

My workaround is to manually mount the host's nvidia-smi binary and the relevant libnvidia-ml.so.1 libcuda.so.1 libraries into the container.

The NVIDIA driver from my host was installed via a .run file. The relevant files are located in /usr/lib64/ and /usr/bin/. Here is the command:

enroot start \
  --root \
  --rw \
  --env LD_LIBRARY_PATH=/usr/lib64 \
  --mount /usr/lib64/libcuda.so.1:/usr/lib64/libcuda.so.1 \
  --mount /usr/lib64/libnvidia-ml.so.1:/usr/lib64/libnvidia-ml.so.1 \
  --mount /usr/bin/nvidia-smi:/usr/bin/nvidia-smi \
  debian+bookworm-slim \
  nvidia-smi

Output example:

Wed Dec 17 15:40:17 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.153.02             Driver Version: 570.153.02     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 D      Off |   00000000:41:00.0 Off |                  Off |
| 30%   26C    P8             15W /  425W |       1MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Applications like PyTorch(with CUDA runtime provided through PyPI packages) can now access the GPU as expected.

Additional Information

Each time I start an Enroot sandbox, there is an error message popping up:

/etc/enroot/hooks.d/10-shadow.sh: line 70: 312531 Broken pipe             yes 2> /dev/null
     312532 Segmentation fault      (core dumped) | pwck -R "${ENROOT_ROOTFS}" "${pwddb#${ENROOT_ROOTFS}}" /etc/shadow > /dev/null 2>&1

This seems unrelated to the NVIDIA driver, but I thought I should mention it in case it is relevant.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions