containerd: Failed to pull image (unexpected commit digest)

I find a way to solve this temporarily.

Just clean the image layer cache folder in ${containerd folder}/io.containerd.content.v1.content/ingest.

Containerd would not clean this cache automatically when some layer data broken.

https://github.com/containerd/containerd/blob/v1.5.2/content/local/store.go#L503

errors.Wrapf on a nil error will return nil, so this caused the error case where ref is incorrect to be misinterpreted as a success case.

Hi @fuweid 😃 I was able to reproduce it locally using toxiproxy .

The issue is: the connection is getting closed. I can reproduce this by using toxiproxy to close the connection after X bytes.

I’m using the following Dockerfile:

FROM ubuntu:20.04

RUN apt-get update && apt-get install -y wget dnsutils
RUN wget -O /usr/local/bin/toxiproxy-cli https://github.com/Shopify/toxiproxy/releases/download/v2.1.4/toxiproxy-cli-linux-amd64
RUN wget -O /usr/local/bin/toxiproxy-server https://github.com/Shopify/toxiproxy/releases/download/v2.1.4/toxiproxy-server-linux-amd64
RUN chmod +x /usr/local/bin/toxiproxy-cli /usr/local/bin/toxiproxy-server

RUN wget -O /tmp/containerd.tar.gz https://github.com/containerd/containerd/releases/download/v1.4.4/containerd-1.4.4-linux-amd64.tar.gz \
  && tar -xvzf /tmp/containerd.tar.gz -C /usr/local/ \
  && rm /tmp/containerd.tar.gz
RUN /usr/bin/mkdir /etc/containerd \
  && containerd config default > /etc/containerd/config.toml
RUN wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.21.0/crictl-v1.21.0-linux-amd64.tar.gz \
  && tar zxvf crictl-v1.21.0-linux-amd64.tar.gz -C /usr/local/bin \
  && rm -f crictl-v1.21.0-linux-amd64.tar.gz

Build and start the image:

docker build -t foo .
docker run -ti --privileged --rm foo

Inside the container start I’m:

• starting containerd
• starting toxiproxy
• configure toxiproxy to close the tcp connection after 1 MB data transfer
• reconfigure /etc/hosts to pull the image through the tcp proxy (via storage.googleapis.com)

containerd &
toxiproxy-server &

IP=$(dig +short storage.googleapis.com | grep -v googlecode | head -n 1)
toxiproxy-cli create registry -l 127.0.0.1:443 -u "$IP:443"
toxiproxy-cli toxic add registry -t limit_data -a bytes=1048576
echo "127.0.0.1 storage.googleapis.com" >> /etc/hosts

crictl -D pull k8s.gcr.io/etcd:3.4.13-0

I hope this helps 😃

Thank you so much! That helped also for me! For reference, in k3s the folder is: /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content/ingest

Until the fix is released and gets rolled out into distributions, I’ll share my workaround. Seems like the ctr command line is able to fix broken image cache layers. So you can run something like:

ctr i pull docker.io/rancher/library-busybox:1.32.1 >/dev/null
ctr i pull docker.io/rancher/library-traefik:2.4.8 >/dev/null
ctr i pull docker.io/library/postgres:11 >/dev/null

Hi @fuweid :

I just used the proxy to simulate the issue which happens somewhere in our network (somebody closing the tcp connection). In our network this seems to always happen if the layer is big and it takes too long to download it from the registry.

So I see 2 possible ways to resolve this:

• This works as designed, (at least for my case) the issue here get’s closed and it is only solvable by fixing whoever is closing the tcp connection to not do that, not having any way to workaround
• Containerd is able to handle this and continue the download (like e.g. wget would be able to)

I’d be fine with both (of course I would be more happy with resolving both sides) 😃

Thank you for your help 👍