containerd: Image pull error: not a directory: unknown

we are facing the same issue with one of our container build with kaniko as well. is there any update on this?

Looks like this may have been fixed in https://github.com/GoogleContainerTools/kaniko/pull/2066. I was seeing the same issue, tried gcr.io/kaniko-project/executor:aad03dc285fbfddd3d6d31637629d0423604129a (which contains the fix), and everything seems to be pulling correctly.

Thanks @qiutongs for the investigation!

So what kaniko does that is a bit suspicous is that it constructs a container image with:

Lower layer:

...
/root/.wh.dirA
...

Upper layer:

/root/dirA/dirB/.wh.fileA
/
/root/
/root/dirA/
/root/dirA/fileB/
...

i.e. an upper layer has a file under a directory that’s been deleted on lower layer, AND the file appears before the parent directories on upper layer.

In the example above, tar.go will complain lstat: /root/dirA/dirB/ not a directory, when calling mkparent on /root/dirA/dirB/.wh.fileA.

It’s good to know that Docker build does not trigger this case (at least not easily). But this case does not seem illegal. If we want to fix it on containerd, I can see two options:

1. Same as what @knight42 proposed in https://github.com/containerd/containerd/issues/4659#issuecomment-731719182, i.e. skip copying up from parent snapshots for whiteout files. This will work but this fix is based on the assumption that whiteout files always appear before regular files.

2. If fs.RootPath() returns lstat not a directory error, we ignore it. But I am not sure whether this could cause other problems.

@AkihiroSuda Hi, I think this is a bug of containerd as the image r0bj/test-image:0.1 seems to be valid, and this is my analysis of the root cause https://github.com/containerd/containerd/issues/4659#issuecomment-731595115. I think this issue might be related to https://github.com/containerd/containerd/pull/3528.

I also have a fix for this problem:

diff --git archive/tar.go archive/tar.go
index 580e6e76f..5aa1eeed2 100644
--- archive/tar.go
+++ archive/tar.go
@@ -228,7 +228,7 @@ func applyNaive(ctx context.Context, root string, tr *tar.Reader, options ApplyO
 			if base == "" {
 				parentPath = filepath.Dir(path)
 			}
-			if err := mkparent(ctx, parentPath, root, options.Parents); err != nil {
+			if err := mkparent(ctx, parentPath, root, options.Parents, strings.HasPrefix(base, whiteoutPrefix)); err != nil {
 				return 0, err
 			}
 		}
@@ -379,7 +379,7 @@ func createTarFile(ctx context.Context, path, extractDir string, hdr *tar.Header
 	return chtimes(path, boundTime(latestTime(hdr.AccessTime, hdr.ModTime)), boundTime(hdr.ModTime))
 }
 
-func mkparent(ctx context.Context, path, root string, parents []string) error {
+func mkparent(ctx context.Context, path, root string, parents []string, isWhiteout bool) error {
 	if dir, err := os.Lstat(path); err == nil {
 		if dir.IsDir() {
 			return nil
@@ -399,7 +399,7 @@ func mkparent(ctx context.Context, path, root string, parents []string) error {
 	}
 
 	if i > len(root)+1 {
-		if err := mkparent(ctx, path[:i-1], root, parents); err != nil {
+		if err := mkparent(ctx, path[:i-1], root, parents, isWhiteout); err != nil {
 			return err
 		}
 	}
@@ -413,6 +413,10 @@ func mkparent(ctx context.Context, path, root string, parents []string) error {
 		return err
 	}
 
+	// The file is deleted, we don't need to create parent dir for it in the lower layers
+	if isWhiteout {
+		return nil
+	}
 	for _, p := range parents {
 		ppath, err := fs.RootPath(p, path[len(root):])
 		if err != nil {

@AkihiroSuda Is this change reasonable?