extensions: [nvidia-container-toolkit=530.41.03-v1.12.1] [talos=v1.4.5] crash on container creation when NVIDIA_DRIVER_CAPABILITIES contains "graphics"

I’m not 100% sure if this is the right place to report the issue, feel free to point me in the right direction if not 😃

My setup:

I am running a 2 node cluster based on talos 1.4.5 and i’ve enabled the nvidia container toolkit extension by following the official talos tutorial (changing all mentions of talos version v1.4.4 to v1.4.5 and using version v0.14.0 of the nvidia device plugin)

The nodes both contain a RTX 2080 GPU

The issue:

All conventional cuda based container workloads i’ve tested run without any problems, nvidia-smi from inside of a container also reports the proper gpu information.

whenever i add graphics to the NVIDIA_DRIVER_CAPABILITIES env variable container creation fails with the following status:

failed to create containerd task: 
failed to create shim task: 
OCI runtime create failed: 
failed to create NVIDIA Container Runtime: 
failed to construct OCI spec modifier: 
failed to construct discoverer: 
failed to create mounts discoverer: 
failed to construct library locator: 
error loading ldcache: 
open /etc/ld.so.cache: no such file or directory: unknown

i am honestly at a loss here, if anyone has any input why this might be happening i would greatly appreciate any help

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 22 (10 by maintainers)

Commits related to this issue

Most upvoted comments

ah, you can just use a regular ubuntu:23.04 and do apt install vulkan-tools, then vulkaninfo --summary and make sure that a real GPU showed up, not llvmpipe (software/CPU rendering engine)

Thank you so much for looking into it:

error loading ldcache error goes from NewLibraryLocator, which is used by NewGraphicsMountsDiscoverer, which is a part of nvidia-container-runtime. As I understand correctly, nvidia-container-runtime gets executed on the host.

Seems I am wrong. I probably confused this with code that gets executed by the runtime inside container (which also uses libc 😅 )

But i’ve a little confusion, if we directly execute ldconfig it should look into the path that is used in talos. I guess we just need to add a custom search path here: https://github.com/NVIDIA/nvidia-container-toolkit/blob/v1.12.1/internal/discover/graphics.go#L75 (didn’t look too deep).

We need to remove --no-opengl-files and --no-install-libglvnd flags during Nvidia driver installation here.

This can be done, it was added just to drop the extension size

After Nvidia driver installation, we can’t move these OpenGL/Vulkan libraries and files to the /etc and /usr/share folders. We can only use /usr/local/share in Talos. I tried to make a patch for this code and changed it to /usr/local/share, but after that nvidia-container-runtime is mounting these files to /usr/local/share inside a container, which is the wrong location for container applications, because they expect /etc and /usr/share directories.

We could add new allowlist paths for these since it’s standard for OpenGL/Vulkan i guess, (explicit like /etc/opengl or something)

You can do a fix, perform some testing and submit a PR to speed up things.

Otherwise we’ll get to it as soon as we can, no ETA is available atm.