cri-o: Latest CRI-O (on 1.25/1.26) failing to install nvidia gpu-operator due to failing to mount subscriptions
What happened?
It looks like the change is specifically the change from using the os.WriteFile function to using the umask.WriteFileIgnoreUmask function on this line: https://github.com/cri-o/cri-o/pull/7774/files#diff-23e01fcec1708a4fa51b3f495b7c7f075070b0a9c5a9195f349efee6d9444d4dR271
crio fails to mount the subscription to the container, as see in these logs (more above):
Failed to mount subscriptions, skipping entry in /usr/share/containers/mounts.conf: saving data to container filesystem
What did you expect to happen?
nvidia gpu-operator (specifically the driver installer) attempts to use the rhel entitlement of the server the container is running on to install required packages.
I expect the install to continue to work.
How can we reproduce it (as minimally and precisely as possible)?
Use container runtime crio-o on version(s) cri-o-1.25.5-10.rhaos4.12.git8724c4d.el8 cri-o-1.26.5-7.rhaos4.13.git692ef91.el8
Try to run a container that will attempt to use the rhel8 entitlement (failure in nvidia gpu-operator use case is dnf install -q -y elfutils-libelf.x86_64 elfutils-libelf-devel.x86_64
Anything else we need to know?
Documented further details with nvidia here: https://github.com/NVIDIA/gpu-operator/issues/680
CRI-O and Kubernetes version
cri-o-1.25.5-10.rhaos4.12.git8724c4d.el8 cri-o-1.26.5-7.rhaos4.13.git692ef91.el8
Kubernetes version 1.25/1.26 respectively
OCP versions 4.12/4.13 respectively
OS version
RHEL8.9
Additional environment details (AWS, VirtualBox, physical, etc.)
Self OCP install can reproduce. We (IBM Cloud) did not release latest cri-o because of this regression.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (11 by maintainers)
Using:
cri-o://1.26.5-10.rhaos4.13.gita08b329.el8and
cri-o://1.25.5-13.1.rhaos4.12.git76343da.el8I am no longer seeing this issue.
Thank you all for the quick fix!
Ah now I got it, thank you for the backports @kwilczynski 🙏
@saschagrunert We need to pick this regression, https://github.com/containers/common/commit/e6ca254a21929b01253073af65e1031280d8029d
The Pull Requests have been merged into the respective release branches, and Red Hat’s QA team confirmed that the reported issues have been fixed. Thus, I will close this issue now. We can reopen it if needed.
Thank you, everyone, for your help!
This also impacts the nvidia-network-operator (really anything that utilizes entitlements in their execution process and depends on those mounts existing for both RHEL and RHCOS)
@KodieGlosserIBM, per what @hasan4791 shared above, the newer versions of CRI-O would include the linked fix.
Such that anything that uses
containers/common(inclusive of) 0.53.0, would not be affected.@hasan4791, I will see about backporting this to the relevant
containers/commonbranches, and then updating our dependency. Bear with me.@saschagrunert for 1.27/1.28 we are using cri-o-1.27.4-2.rhaos4.14.gita200bb4.el8.x86_64.rpm cri-o-1.28.3-14.rhaos4.15.git33aabd8.el8.x86_64.rpm
And are not seeing issues. For 1.29 we are not yet testing with.