cri-o: Latest CRI-O (on 1.25/1.26) failing to install nvidia gpu-operator due to failing to mount subscriptions

What happened?

It looks like the change is specifically the change from using the os.WriteFile function to using the umask.WriteFileIgnoreUmask function on this line: https://github.com/cri-o/cri-o/pull/7774/files#diff-23e01fcec1708a4fa51b3f495b7c7f075070b0a9c5a9195f349efee6d9444d4dR271

crio fails to mount the subscription to the container, as see in these logs (more above):

Failed to mount subscriptions, skipping entry in /usr/share/containers/mounts.conf: saving data to container filesystem

What did you expect to happen?

nvidia gpu-operator (specifically the driver installer) attempts to use the rhel entitlement of the server the container is running on to install required packages.

I expect the install to continue to work.

How can we reproduce it (as minimally and precisely as possible)?

Use container runtime crio-o on version(s) cri-o-1.25.5-10.rhaos4.12.git8724c4d.el8 cri-o-1.26.5-7.rhaos4.13.git692ef91.el8

Try to run a container that will attempt to use the rhel8 entitlement (failure in nvidia gpu-operator use case is dnf install -q -y elfutils-libelf.x86_64 elfutils-libelf-devel.x86_64

Anything else we need to know?

Documented further details with nvidia here: https://github.com/NVIDIA/gpu-operator/issues/680

CRI-O and Kubernetes version

cri-o-1.25.5-10.rhaos4.12.git8724c4d.el8 cri-o-1.26.5-7.rhaos4.13.git692ef91.el8 Kubernetes version 1.25/1.26 respectively OCP versions 4.12/4.13 respectively

OS version

RHEL8.9

Additional environment details (AWS, VirtualBox, physical, etc.)

Self OCP install can reproduce. We (IBM Cloud) did not release latest cri-o because of this regression.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 18 (11 by maintainers)

Most upvoted comments

Using: cri-o://1.26.5-10.rhaos4.13.gita08b329.el8

and

cri-o://1.25.5-13.1.rhaos4.12.git76343da.el8

I am no longer seeing this issue.

Thank you all for the quick fix!

Ah now I got it, thank you for the backports @kwilczynski 🙏

The Pull Requests have been merged into the respective release branches, and Red Hat’s QA team confirmed that the reported issues have been fixed. Thus, I will close this issue now. We can reopen it if needed.

Thank you, everyone, for your help!

This also impacts the nvidia-network-operator (really anything that utilizes entitlements in their execution process and depends on those mounts existing for both RHEL and RHCOS)

@KodieGlosserIBM, per what @hasan4791 shared above, the newer versions of CRI-O would include the linked fix.

Such that anything that uses containers/common (inclusive of) 0.53.0, would not be affected.

@hasan4791, I will see about backporting this to the relevant containers/common branches, and then updating our dependency. Bear with me.

@saschagrunert for 1.27/1.28 we are using cri-o-1.27.4-2.rhaos4.14.gita200bb4.el8.x86_64.rpm cri-o-1.28.3-14.rhaos4.15.git33aabd8.el8.x86_64.rpm

And are not seeing issues. For 1.29 we are not yet testing with.