kata-containers: cri-o does not work in kata-containers 2.2.0: Failed to create pod sandbox: rpc error: code = Unknown desc = dial unix /run/containerd/s/a554844d06959beb34922f4e5411c5ff380d2f72e2d9505d406109cebcc8bc5d: connection refused

Link https://github.com/kubernetes/kubernetes/issues/104929

Description of problem

When I create a pod through kata in kubernetes configured with cri-o, the pod is not created.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
      securityContext:
        privileged: true
  runtimeClassName: kata-qemu

Sep 11 02:50:10 k8s-ci-node-05102 kubelet[751]: /run/containerd/s/a613e1e8bdab5137886078fbc47bd0159066e7b7ed19f5f4a6620cb5343dc4aa: connect: connection refused
Sep 11 02:50:10 k8s-ci-node-05102 kubelet[751]: /run/containerd/s/a613e1e8bdab5137886078fbc47bd0159066e7b7ed19f5f4a6620cb5343dc4aa: connect: connection refused
Sep 11 02:50:10 k8s-ci-node-05102 kubelet[751]: /run/containerd/s/a613e1e8bdab5137886078fbc47bd0159066e7b7ed19f5f4a6620cb5343dc4aa: connect: connection refused
Sep 11 02:50:10 k8s-ci-node-05102 kubelet[751]: E0911 02:50:10.986267     751 pod_workers.go:191] Error syncing pod 4796973e-8f3a-4830-9101-1f197541da71 ("nginx_default(4796973e-8f3a-4830-9101-1f197541da71)"), skipping: failed to "CreatePodSandbox" for "nginx_default(4796973e-8f3a-4830-9101-1f197541da71)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx_default(4796973e-8f3a-4830-9101-1f197541da71)\" failed: rpc error: code = Unknown desc = dial unix \x00/run/containerd/s/a613e1e8bdab5137886078fbc47bd0159066e7b7ed19f5f4a6620cb5343dc4aa: connect: connection refused"

More informations

/usr/local/bin/containerd-shim-kata-v2:

#!/bin/sh
/snap/bin/kata-containers.shim "$@"

kata-collect

https://gist.github.com/jclab-joseph/7444ca1f9a3d23f285a48d9635a230ae

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 2
Comments: 41 (14 by maintainers)

Most upvoted comments

@git-noise I believe this is crio related. I’ve not traced it to a specific commit but I saw this same problem with 1.20.7 and if I change only the crio binary to 1.21.6 it goes away. Note that I still see the sockets created under /run/containerd/s but there’s no longer an error.

Sorry to not give you a root cause but felt it was worth sharing. I saw the same \x00/run/containerd/s/ in my logfile, maybe there’s an extra prefix char there in crio

mark-au on May 10, 2022

@fgiudici Yes of course, I am using Ubuntu and everything was installed manually using the indicated repositories for CRI-O or distribution repositories via apt - and snap for kata. At least now we know it seems to be some configuration issue. As I mentioned, neither CRI-O or kata configuration files are referencing containerd anymore - and its not even installed, so I could not figure out where some ‘containerd’ config may have lingered - an why it appeared from 2.2 only.

I’ll re-deploy something then and see if it solves everything.

thanks!

git-noise on Mar 17, 2022

Thanks @Wang-Kai, I am afraid it is still not working for me - although I am using crio version 1.20.6. Are you deploying on a bare metal or virtualized cluster?

@fidencio at this point I will try to redeploy on another cluster, possibly with containerd, to try to pinpoint if it’s cri-o, containerd, virtualization within VM or something else related. I am still a bit confused why we are seeing /run/containerd/ related path in a crio-o installation.

I deployed it on a bare metal.

Wang-Kai on Mar 3, 2022

Thanks a lot for the info. I’ll follow the main branch, build and report if I can - it may not be immediate though.

git-noise on Nov 3, 2021