containerd: Image pull error: not a directory: unknown
Description
crictl pull finishes with error:
# crictl pull r0bj/test-image:0.1
FATA[0002] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/r0bj/test-image:0.1": unpack: failed to extract layer sha256:9c859ee9ae8224ac8800320b0e14ebffed2f4c82d4a37c37130e1672f9d6c26b: lstat /data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/10382/fs/root/.gem/specs/api.rubygems.org%443: not a directory: unknown
Dockerfile used to build image:
FROM fluent/fluentd:v1.9.3-1.0
USER root
RUN apk add --no-cache --update --virtual .build-deps \
sudo build-base ruby-dev \
&& sudo gem install \
fluent-plugin-nsq:0.0.6 \
fluent-plugin-kubernetes_metadata_filter:2.4.1 \
fluent-plugin-record-modifier:2.1.0 \
fluent-plugin-prometheus:1.7.3 \
fluent-plugin-throttle:0.0.4 \
fluent-plugin-multi-format-parser:1.0.0 \
&& sudo gem sources --clear-all \
&& apk del .build-deps \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
RUN gem install nsq-ruby:2.3.1 && gem uninstall -I nsq-ruby --version 1.7.0
Describe the results you received:
# crictl pull r0bj/test-image:0.1
FATA[0002] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/r0bj/test-image:0.1": unpack: failed to extract layer sha256:9c859ee9ae8224ac8800320b0e14ebffed2f4c82d4a37c37130e1672f9d6c26b: lstat /data/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/10382/fs/root/.gem/specs/api.rubygems.org%443: not a directory: unknown
Describe the results you expected:
Image is pulled withour error.
Output of containerd --version
:
containerd containerd.io 1.3.7 8fba4e9a7d01810a393d5d25a3621dc101981175
Any other relevant information:
Image is pulled successfully using docker:
$ docker pull r0bj/test-image:0.1
0.1: Pulling from r0bj/test-image
9123ac7c32f7: Pull complete
78b3236c5266: Pull complete
76b97e7b035b: Pull complete
d19a0d92c227: Pull complete
e5eafb0c1052: Pull complete
cff2046d76ca: Pull complete
a12add9f9520: Pull complete
Digest: sha256:7871299f8dddcd1ee6bbfb5c522b8146b3a067d2de5394e327b6212453e7c8fe
Status: Downloaded newer image for r0bj/test-image:0.1
docker.io/r0bj/test-image:0.1
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (13 by maintainers)
we are facing the same issue with one of our container build with kaniko as well. is there any update on this?
Looks like this may have been fixed in https://github.com/GoogleContainerTools/kaniko/pull/2066. I was seeing the same issue, tried
gcr.io/kaniko-project/executor:aad03dc285fbfddd3d6d31637629d0423604129a
(which contains the fix), and everything seems to be pulling correctly.Thanks @qiutongs for the investigation!
So what kaniko does that is a bit suspicous is that it constructs a container image with:
Lower layer:
Upper layer:
i.e. an upper layer has a file under a directory that’s been deleted on lower layer, AND the file appears before the parent directories on upper layer.
In the example above, tar.go will complain
lstat: /root/dirA/dirB/ not a directory
, when callingmkparent
on/root/dirA/dirB/.wh.fileA
.It’s good to know that Docker build does not trigger this case (at least not easily). But this case does not seem illegal. If we want to fix it on containerd, I can see two options:
Same as what @knight42 proposed in https://github.com/containerd/containerd/issues/4659#issuecomment-731719182, i.e. skip copying up from parent snapshots for whiteout files. This will work but this fix is based on the assumption that whiteout files always appear before regular files.
If fs.RootPath() returns
lstat not a directory
error, we ignore it. But I am not sure whether this could cause other problems.@AkihiroSuda Hi, I think this is a bug of containerd as the image
r0bj/test-image:0.1
seems to be valid, and this is my analysis of the root cause https://github.com/containerd/containerd/issues/4659#issuecomment-731595115. I think this issue might be related to https://github.com/containerd/containerd/pull/3528.I also have a fix for this problem:
@AkihiroSuda Is this change reasonable?