cri-o: Pulling container images fails on CentOS Stream 8
What happened?
Pulling images fails due to a unknown key in /etc/containers/policy.json on Centos Stream 8 with cri-o 1.24.2 installed.
[root@localhost ~]# crictl pull k8s.gcr.io/kube-apiserver:v1.24.4
FATA[0000] pulling image: rpc error: code = Unknown desc = invalid policy in "/etc/containers/policy.json": Unknown key "keyPaths"
The same image can be pulled successfully using podman
[root@localhost brian]# podman pull k8s.gcr.io/kube-apiserver:v1.24.4
Trying to pull k8s.gcr.io/kube-apiserver:v1.24.4...
Getting image source signatures
Copying blob f5bb0a2b916a done
Copying blob b9f88661235d done
Copying blob cca57b588e6e done
Copying config 6cab9d1bed done
Writing manifest to image destination
Storing signatures
6cab9d1bed1be49c215505c1a438ce0af66eb54b4e95f06e52037fcd36631f3d
This issue looks to have been introduced by updates in the latest containers-common rpm (containers-common-1-40.module_el8.7.0+1196+721f4eb0.x86_64)
Pulling images succeed after downgrading containers-common
[root@localhost ~]# rpm -qa | grep containers-common
containers-common-1-40.module_el8.7.0+1196+721f4eb0.x86_64
[root@localhost ~]# cat /etc/containers/policy.json | grep keyPath
"keyPaths": ["/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release", "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"]
"keyPaths": ["/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release", "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta"]
[root@localhost ~]# crictl pull quay.io/kubevirtci/fedora:36-2208010931
FATA[0003] pulling image: rpc error: code = Unknown desc = invalid policy in "/etc/containers/policy.json": Unknown key "keyPaths"
[root@localhost ~]# dnf downgrade containers-common
....
...
[root@localhost ~]# rpm -qa | grep containers-common
containers-common-1-23.module_el8.7.0+1106+45480ee0.x86_64
[root@localhost ~]# cat /etc/containers/policy.json | grep keyPath
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
"keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
[root@localhost ~]# crictl pull quay.io/kubevirtci/fedora:36-2208010931
Image is up to date for quay.io/kubevirtci/fedora@sha256:486fd5578f93fbc57a519e34ad4b7cac927c3f8a95409baedf0c19e9f287c207
What did you expect to happen?
Images are pulled successfully
How can we reproduce it (as minimally and precisely as possible)?
- Install Centos Stream 8
- Run a dnf upgrade
dnf upgrade - Install cri-o following the documented install procedure for
CentOS_8_Stream - Install cri-tools to get crictl
dnf install -y cri-tools - Enable crio and start crio
systemctl enable crio --now - Try to pull an image
[root@localhost ~]# crictl pull quay.io/kubevirtci/fedora:36-2208010931
FATA[0003] pulling image: rpc error: code = Unknown desc = invalid policy in "/etc/containers/policy.json": Unknown key "keyPaths"
Anything else we need to know?
No response
CRI-O and Kubernetes version
$ crio --version
[root@localhost ~]# crio --version
WARN[0000] Failed to decode the keys ["network.network_backend"] from "/usr/share/containers/containers.conf".
crio version 1.24.2
Version: 1.24.2
GitCommit: bd548b04f78a30e1e9d7c17162714edd50edd6ca
GitTreeState: clean
BuildDate: 2022-08-09T18:58:47Z
GoVersion: go1.18.2
Compiler: gc
Platform: linux/amd64
Linkmode: dynamic
BuildTags: exclude_graphdriver_devicemapper, seccomp
SeccompEnabled: true
AppArmorEnabled: false
$ kubectl --version
NOT INSTALLED
OS version
# On Linux:
$ cat /etc/os-release
[root@localhost ~]# cat /etc/os-release
NAME="CentOS Stream"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Stream 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
$ uname -a
[root@localhost ~]# uname -a
Linux localhost.localdomain 4.18.0-408.el8.x86_64 #1 SMP Mon Jul 18 17:42:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Additional environment details (AWS, VirtualBox, physical, etc.)
Tested in a Centos Stream 8 VM
Also seen in some automation testing carried out by Kubevirt For example: https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirtci/857/check-provision-k8s-1.24/1565236465217048576
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 12
- Comments: 16 (6 by maintainers)
Thank you for reporting on this, I was about to as I had been dealing with this as well.
To fix this for now I created the following ansible step:
Noticed changes in this commit - https://gitlab.com/redhat/centos-stream/rpms/containers-common/-/commit/04645c4a84442da3324eea8f6538a5768e69919a
Similar issue on RHEL 8.6 with
containers-common-2:1-43. Had to editetc/containers/policy.json. Set keypath as @ccravens instructed. Edited keypath{ "default": [ { "type": "insecureAcceptAnything" } ], "transports": { "docker": { "registry.access.redhat.com": [ { "type": "signedBy", "keyType": "GPGKeys", "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release" } ], "registry.redhat.io": [ { "type": "signedBy", "keyType": "GPGKeys", "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release" } ] }, "docker-daemon": { "": [ { "type": "insecureAcceptAnything" } ] } } }Similar Problem on k8s version : v1.25.4 :
solution
that’s it. the kube-proxy pod successfully created.