cri-o: Policy Signature Verification Fails

What happened?

Using a policy define like this, when crio starts a container, the signature verification fails with:

Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.904125667Z" level=debug msg="Response: &RunPodSandboxResponse{PodSandboxId:327f5ecea3068153344a14a569c254875c05e64daf45769fced1503f9a99066f,}" file="otel-collector/interceptors.go:75" id=a403abc5-f746-4ace-a453-9d6244b2329c name=/runtime.v1.RuntimeService/RunPodSandbox
Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.905114611Z" level=debug msg="Response error: checking signature of \"registry.k8s.io/kube-controller-manager:v1.31.1\": verifying signatures: Invalid crypto engine" file="otel-collector/interceptors.go:72" id=86731c11-5ad3-442b-bca6-cb2ab551cfa5 name=/runtime.v1.RuntimeService/CreateContainer
Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.907250611Z" level=debug msg="Request: &PodSandboxStatusRequest{PodSandboxId:327f5ecea3068153344a14a569c254875c05e64daf45769fced1503f9a99066f,Verbose:false,}" file="otel-collector/interceptors.go:63" id=0d2927fa-4a43-4607-b164-9576e13e21f6 name=/runtime.v1.RuntimeService/PodSandboxStatus
Oct 09 15:39:26 HOST crio[2094]: time="2024-10-09 15:39:26.907396114Z" level=debug msg="Response: &PodSandboxStatusResponse{Status:&PodSandboxStatus{Id:327f5ecea3068153344a14a569c254875c05e64daf45769fced1503f9a99066f,Metadata:&PodSandboxMetadata{Name:kube-scheduler-HOST,Uid:d1b799f8aa6f6a039bd87cb5c290988a,Namespace:kube-system,Attempt:0,},State:SANDBOX_READY,CreatedAt:1728488366884356609,Network:&PodSandboxNetworkStatus{Ip:,AdditionalIps:[]*PodIP{},},Linux:&LinuxPodSandboxStatus{Namespaces:&Namespace{Options:&NamespaceOption{Network:NODE,Pid:CONTAINER,Ipc:POD,TargetId:,UsernsOptions:nil,},},},Labels:map[string]string{component: kube-scheduler,io.kubernetes.container.name: POD,io.kubernetes.pod.name: kube-scheduler-HOST,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: d1b799f8aa6f6a039bd87cb5c290988a,tier: control-plane,},Annotations:map[string]string{kubernetes.io/config.hash: d1b799f8aa6f6a039bd87cb5c290988a,kubernetes.io/config.seen: 2024-10-09T15:39:25.956572130Z,kubernetes.io/config.source: file,},RuntimeHandler:,},Info:map[string]string{},ContainersStatuses:[]*ContainerStatus{},Timestamp:0,}" file="otel-collector/interceptors.go:75" id=0d2927fa-4a43-4607-b164-9576e13e21f6 name=/runtime.v1.RuntimeService/PodSandboxStatus
 "docker": {
             "registry.k8s.io": [{
                 "type": "signedBy",
                 "keyType": "GPGKeys",
                 "keyPath": "/home/image_signing/key.gpg"
              }],
            "docker.io": [{
                "type": "signedBy",
                "keyType": "GPGKeys",
                "keyPath": "/home/image_signing/key.gpg"
              }]
        }

What did you expect to happen?

The image signatures to be verified.

How can we reproduce it (as minimally and precisely as possible)?

Run image signature verification on pod start. This is on a s390x machine.

Anything else we need to know?

gpg version is 2.3.3

((root@ZCPA1))>gpgconf gpg:OpenPGP:/usr/bin/gpg gpgsm:S/MIME:/usr/bin/gpgsm keyboxd:Public Keys:/usr/libexec/keyboxd gpg-agent:Private Keys:/usr/bin/gpg-agent scdaemon:Smartcards:/usr/libexec/scdaemon dirmngr:Network:/usr/bin/dirmngr pinentry:Passphrase Entry:/usr/bin/pinentry

/usr/lib64/libgpgme.so.11.24.1

I’m able to create a test program in both C and go that makes a new context, sets the engine info, and imports the keys. This is dynamically linked to the shared object, so that might be a difference from crio.

CRI-O and Kubernetes version

$ crio --version
crio version 1.31.1
   GitCommit:      1328c762815533fb201b0c4362741bc9ab2ce2e6
   GitCommitDate:  2024-10-01T19:38:55Z
   GitTreeState:   dirty
   BuildDate:      1970-01-01T00:00:00Z
   GoVersion:      go1.22.5
   Compiler:       gc
   Platform:       linux/s390x
   Linkmode:       static
   BuildTags:
     static
     netgo
     osusergo
     exclude_graphdriver_btrfs
     seccomp
     apparmor
     selinux
     exclude_graphdriver_devicemapper
   LDFlags:          unknown
   SeccompEnabled:   true
   AppArmorEnabled:  false

$ kubectl version --output=json
{
  "clientVersion": {
    "major": "1",
    "minor": "31",
    "gitVersion": "v1.31.1",
    "gitCommit": "948afe5ca072329a73c8e79ed5938717a5cb3d21",
    "gitTreeState": "clean",
    "buildDate": "2024-09-11T21:28:00Z",
    "goVersion": "go1.22.6",
    "compiler": "gc",
    "platform": "linux/s390x"
  },
  "kustomizeVersion": "v5.4.2"
}

OS version

# On Linux:
$ cat /etc/os-release
# RHEL 9
$ uname -a
Linux HOST 5.14.0-362.24.1.el9.s390x #1 SMP Mon Jul 1 09:04:29 CDT 2024 s390x s390x s390x GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (6 by maintainers)

Most upvoted comments

Thank you @saschagrunert! I think that fixed the gpgme issue. I’m running into an error starting pods, but I suspect that’s just running k8s 1.31 while these binaries were built for 1.32.

Edit: I built your changes against 1.31 statically and everything came up great.

@saschagrunert Thank you… You updated this, just as I was about to ask you if we could override or remove the --enable-fixed-path in the gpgme configureFlags.

I am sure @MaryJulian17 will try this change as soon as she gets in this morning.

Thank you!