extensions: [nvidia-container-toolkit=530.41.03-v1.12.1] [talos=v1.4.5] crash on container creation when NVIDIA_DRIVER_CAPABILITIES contains "graphics"
I’m not 100% sure if this is the right place to report the issue, feel free to point me in the right direction if not 😃
My setup:
I am running a 2 node cluster based on talos 1.4.5 and i’ve enabled the nvidia container toolkit extension by following the official talos tutorial (changing all mentions of talos version v1.4.4 to v1.4.5 and using version v0.14.0 of the nvidia device plugin)
The nodes both contain a RTX 2080 GPU
The issue:
All conventional cuda based container workloads i’ve tested run without any problems, nvidia-smi from inside of a container also reports the proper gpu information.
whenever i add graphics to the NVIDIA_DRIVER_CAPABILITIES env variable container creation fails with the following status:
failed to create containerd task:
failed to create shim task:
OCI runtime create failed:
failed to create NVIDIA Container Runtime:
failed to construct OCI spec modifier:
failed to construct discoverer:
failed to create mounts discoverer:
failed to construct library locator:
error loading ldcache:
open /etc/ld.so.cache: no such file or directory: unknown
i am honestly at a loss here, if anyone has any input why this might be happening i would greatly appreciate any help
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 22 (10 by maintainers)
Commits related to this issue
- feat: update extension spec allowlist for opengl NVIDIA OpenGL/Vulkan files needs to be explicitly in the new locations added. Ref: https://github.com/NVIDIA/nvidia-container-toolkit/blob/v1.13.5/in... — committed to frezbo/extensions by frezbo 3 years ago
- feat: update extension spec allowlist for opengl NVIDIA OpenGL/Vulkan files are super hard-coded. Ref: https://github.com/siderolabs/extensions/pull/191 Fixes: https://github.com/siderolabs/extensi... — committed to frezbo/talos by frezbo 3 years ago
- feat: use wolfi as base for nvidia Use wolfi base as toolchain for NVIDIA build. This removes a lot of hacks and patches we maintain. Fixes: #171 Fixes: https://github.com/siderolabs/pkgs/issues/720... — committed to frezbo/extensions by frezbo 3 years ago
- feat: use wolfi as base for nvidia Use wolfi base as toolchain for NVIDIA build. This removes a lot of hacks and patches we maintain. Fixes: #171 Fixes: https://github.com/siderolabs/pkgs/issues/720... — committed to frezbo/extensions by frezbo 3 years ago
- feat: use wolfi as base for nvidia Use wolfi base as toolchain for NVIDIA build. This removes a lot of hacks and patches we maintain. Fixes: #171 Fixes: https://github.com/siderolabs/pkgs/issues/720... — committed to frezbo/extensions by frezbo 3 years ago
- feat: use wolfi as base for nvidia Use wolfi base as toolchain for NVIDIA build. This removes a lot of hacks and patches we maintain. Fixes: #171 Fixes: https://github.com/siderolabs/pkgs/issues/720... — committed to frezbo/extensions by frezbo 3 years ago
- feat: use wolfi as base for nvidia Use wolfi base as toolchain for NVIDIA build. This removes a lot of hacks and patches we maintain. Fixes: #171 Fixes: https://github.com/siderolabs/pkgs/issues/720... — committed to frezbo/extensions by frezbo 3 years ago
- feat: use wolfi as base for nvidia Use wolfi base as toolchain for NVIDIA build. This removes a lot of hacks and patches we maintain. Fixes: #171 Fixes: https://github.com/siderolabs/pkgs/issues/720... — committed to frezbo/extensions by frezbo 3 years ago
ah, you can just use a regular
ubuntu:23.04and doapt install vulkan-tools, thenvulkaninfo --summaryand make sure that a real GPU showed up, not llvmpipe (software/CPU rendering engine)Thank you so much for looking into it:
Seems I am wrong. I probably confused this with code that gets executed by the runtime inside container (which also uses libc 😅 )
But i’ve a little confusion, if we directly execute
ldconfigit should look into the path that is used in talos. I guess we just need to add a custom search path here: https://github.com/NVIDIA/nvidia-container-toolkit/blob/v1.12.1/internal/discover/graphics.go#L75 (didn’t look too deep).This can be done, it was added just to drop the extension size
We could add new allowlist paths for these since it’s standard for OpenGL/Vulkan i guess, (explicit like
/etc/openglor something)You can do a fix, perform some testing and submit a PR to speed up things.
Otherwise we’ll get to it as soon as we can, no ETA is available atm.