cri-o: Policy Signature Verification Fails
What happened?
Using a policy define like this, when crio starts a container, the signature verification fails with:
Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.904125667Z" level=debug msg="Response: &RunPodSandboxResponse{PodSandboxId:327f5ecea3068153344a14a569c254875c05e64daf45769fced1503f9a99066f,}" file="otel-collector/interceptors.go:75" id=a403abc5-f746-4ace-a453-9d6244b2329c name=/runtime.v1.RuntimeService/RunPodSandbox
Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.905114611Z" level=debug msg="Response error: checking signature of \"registry.k8s.io/kube-controller-manager:v1.31.1\": verifying signatures: Invalid crypto engine" file="otel-collector/interceptors.go:72" id=86731c11-5ad3-442b-bca6-cb2ab551cfa5 name=/runtime.v1.RuntimeService/CreateContainer
Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.907250611Z" level=debug msg="Request: &PodSandboxStatusRequest{PodSandboxId:327f5ecea3068153344a14a569c254875c05e64daf45769fced1503f9a99066f,Verbose:false,}" file="otel-collector/interceptors.go:63" id=0d2927fa-4a43-4607-b164-9576e13e21f6 name=/runtime.v1.RuntimeService/PodSandboxStatus
Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.907396114Z" level=debug msg="Response: &PodSandboxStatusResponse{Status:&PodSandboxStatus{Id:327f5ecea3068153344a14a569c254875c05e64daf45769fced1503f9a99066f,Metadata:&PodSandboxMetadata{Name:kube-scheduler-HOST,Uid:d1b799f8aa6f6a039bd87cb5c290988a,Namespace:kube-system,Attempt:0,},State:SANDBOX_READY,CreatedAt:1728488366884356609,Network:&PodSandboxNetworkStatus{Ip:,AdditionalIps:[]*PodIP{},},Linux:&LinuxPodSandboxStatus{Namespaces:&Namespace{Options:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,TargetId:,UsernsOptions:nil,},},},Labels:map[string]string{component: kube-scheduler,io.kubernetes.container.name: POD,io.kubernetes.pod.name: kube-scheduler-HOST,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: d1b799f8aa6f6a039bd87cb5c290988a,tier: control-plane,},Annotations:map[string]string{kubernetes.io/config.hash: d1b799f8aa6f6a039bd87cb5c290988a,kubernetes.io/config.seen: 2024-10-09T15:39:25.956572130Z,kubernetes.io/config.source: file,},RuntimeHandler:,},Info:map[string]string{},ContainersStatuses:[]*ContainerStatus{},Timestamp:0,}" file="otel-collector/interceptors.go:75" id=0d2927fa-4a43-4607-b164-9576e13e21f6 name=/runtime.v1.RuntimeService/PodSandboxStatus
"docker": {
"registry.k8s.io": [{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/home/image_signing/key.gpg"
}],
"docker.io": [{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/home/image_signing/key.gpg"
}]
}
What did you expect to happen?
The image signatures to be verified.
How can we reproduce it (as minimally and precisely as possible)?
Run image signature verification on pod start. This is on a s390x machine.
Anything else we need to know?
gpg version is 2.3.3
((root@ZCPA1))>gpgconf gpg:OpenPGP:/usr/bin/gpg gpgsm:S/MIME:/usr/bin/gpgsm keyboxd:Public Keys:/usr/libexec/keyboxd gpg-agent:Private Keys:/usr/bin/gpg-agent scdaemon:Smartcards:/usr/libexec/scdaemon dirmngr:Network:/usr/bin/dirmngr pinentry:Passphrase Entry:/usr/bin/pinentry
/usr/lib64/libgpgme.so.11.24.1
I’m able to create a test program in both C and go that makes a new context, sets the engine info, and imports the keys. This is dynamically linked to the shared object, so that might be a difference from crio.
CRI-O and Kubernetes version
$ crio --version
crio version 1.31.1
GitCommit: 1328c762815533fb201b0c4362741bc9ab2ce2e6
GitCommitDate: 2024-10-01T19:38:55Z
GitTreeState: dirty
BuildDate: 1970-01-01T00:00:00Z
GoVersion: go1.22.5
Compiler: gc
Platform: linux/s390x
Linkmode: static
BuildTags:
static
netgo
osusergo
exclude_graphdriver_btrfs
seccomp
apparmor
selinux
exclude_graphdriver_devicemapper
LDFlags: unknown
SeccompEnabled: true
AppArmorEnabled: false
$ kubectl version --output=json
{
"clientVersion": {
"major": "1",
"minor": "31",
"gitVersion": "v1.31.1",
"gitCommit": "948afe5ca072329a73c8e79ed5938717a5cb3d21",
"gitTreeState": "clean",
"buildDate": "2024-09-11T21:28:00Z",
"goVersion": "go1.22.6",
"compiler": "gc",
"platform": "linux/s390x"
},
"kustomizeVersion": "v5.4.2"
}
OS version
# On Linux:
$ cat /etc/os-release
# RHEL 9
$ uname -a
Linux HOST 5.14.0-362.24.1.el9.s390x #1 SMP Mon Jul 1 09:04:29 CDT 2024 s390x s390x s390x GNU/Linux
Additional environment details (AWS, VirtualBox, physical, etc.)
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 19 (6 by maintainers)
Thank you @saschagrunert! I think that fixed the gpgme issue. I’m running into an error starting pods, but I suspect that’s just running k8s 1.31 while these binaries were built for 1.32.
Edit: I built your changes against 1.31 statically and everything came up great.
@RedHead17 here we go: https://github.com/cri-o/cri-o/pull/8745
@saschagrunert Thank you… You updated this, just as I was about to ask you if we could override or remove the
--enable-fixed-pathin thegpgmeconfigureFlags.I am sure @MaryJulian17 will try this change as soon as she gets in this morning.
Thank you!