cri-tools: unknown service runtime.v1alpha2.ImageService

What happened:

$ crictl ps
FATA[0000] listing containers: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
$ crictl pull quay.io/calico/node:v3.16.5
FATA[0000] pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService
$ crictl version
FATA[0000] getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService
$ crictl -D -r unix:///run/containerd/containerd.sock info
DEBU[0000] get runtime connection                       
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '30s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] StatusRequest: &StatusRequest{Verbose:true,} 
DEBU[0000] StatusResponse: nil                          
FATA[0000] getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

System Info

$ crictl --version
crictl version v1.19.0
$ kubelet --version
Kubernetes v1.19.6
$ /usr/bin/containerd --version
containerd containerd.io 1.3.9 ea765aba0d05254012b0b9e595e995c09186427f
$ docker --version
Docker version 19.03.14, build 5eb3275d40
$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)
uname -a
Linux 5.3.1-1.el7.elrepo.x86_64 #1 SMP Sat Sep 21 09:44:09 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

What I have tried

add “cri-containerd”: true to /etc/docker/daemon.json -> no effect
remove line “disabled_plugins = [“cri”]” from /etc/containerd/config.toml + systemctl restart containerd -> no effect
containerd config default > /etc/containerd/config.toml ; systemctl restart containerd -> no effect

Anything else we need to know

Context and discovery : This error message appeared when running Kubespray Ansible playbook, so I tried to run the command manually. The error in Kubespray :

TASK [download_container | Download image if required] ************************************************************************************************
fatal: [kube-poc-compute1 -> 10.150.233.51]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/crictl", "pull", "quay.io/calico/node:v3.16.5"], "delta": "0:00:00.010535", "end": "2020-12-29 17:22:09.893943", "msg": "non-zero return code", "rc": 1, "start": "2020-12-29 17:22:09.883408", "stderr": "time=\"2020-12-29T17:22:09+01:00\" level=fatal msg=\"pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService\"", "stderr_lines": ["time=\"2020-12-29T17:22:09+01:00\" level=fatal msg=\"pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService\""], "stdout": "", "stdout_lines": []}

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 25 (7 by maintainers)

Most upvoted comments

@ledroide this is exactly your containerd cgroup driver is not equal with kubelet cgroup driver.

check your containerd cgroup driver

if grep "SystemdCgroup = true"  /etc/containerd/config.toml &> /dev/nul; then  
  driver=systemd
else
  driver=cgroupfs
fi
echo "driver is ${driver}"

if systemd . you should add a env file like :

[root@k8s-dev ~]# cat /etc/systemd/system/kubelet.service.d/containerd.conf 
[Service]
Environment="KUBELET_EXTRA_ARGS=--runtime-request-timeout=15m --image-service-endpoint=unix:///run/containerd/containerd.sock  --cgroup-driver=systemd"

or the easy way to do is

$ mv /etc/containerd/config.toml /etc/containerd/config.toml.bak
$ systemctl restart containerd

may this help you.

+77

oldthreefeng on Sep 2, 2022

Comment disabled_plugins = ["cri"] if exist in config.toml .

+22

shenshouer on Jun 2, 2021

as was stated above Comment out the line disabled_plugins = ["cri"] if that line is in your config.toml. An earlier version of docker installed a version of containerd and that came with a minimal config.toml with cri disabled. Thus with cri disabled you can connect to containerd but you will not find any of the cri services.

+11

mikebrow on Oct 11, 2021

Comment disabled_plugins = ["cri"] if exist in config.toml .

I commented that line in /etc/containerd/config.toml and then running systemctl restart containerd worked for me.

manojkumarragupathi on May 6, 2022

@ledroide this is exactly your containerd cgroup driver is not equal with kubelet cgroup driver.

check your containerd cgroup driver
if grep "SystemdCgroup = true"  /etc/containerd/config.toml &> /dev/nul; then  
  driver=systemd
else
  driver=cgroupfs
fi
echo "driver is ${driver}"
if systemd . you should add a env file like :
[root@k8s-dev ~]# cat /etc/systemd/system/kubelet.service.d/containerd.conf 
[Service]
Environment="KUBELET_EXTRA_ARGS=--runtime-request-timeout=15m --image-service-endpoint=unix:///run/containerd/containerd.sock  --cgroup-driver=systemd"
or the easy way to do is
$ rm /etc/containerd/config.toml
$ systemctl restart containerd
may this help you.

Not fixable in my case. Does anyone know any other things that cause this?

gjkim42 on Feb 5, 2021

Facing an same issue with new version of Ubuntu 22.04 and k8s - 1.24.7 with command : kubeadm config images pull and register: pre_pull_kube_images

From the comment at the end here https://computingforgeeks.com/install-kubernetes-cluster-ubuntu-jammy/

... the property mentioned here should be *false* ->

[plugins.”io.containerd.grpc.v1.cri”]
plugins.cri.systemd_cgroup = false

The one that should be true is ->
[plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options]
SystemdCgroup = true

nickolaev on Nov 4, 2022