kaniko: error removing var/run to make way for new symlink: unlinkat

Actual behavior Can not build the image, was working and suddenly was not.

Expected behavior Be able to build the image.

To Reproduce File system:

issue
├── context
│   ├── builder.ash
│   └── Dockerfile
└── index.bash

1 directory, 3 files
  • Create an issue directory
  • Create the following index.bash file under the issue directory:
#!/bin/bash

hereDir=`dirname $0 | while read a; do cd $a && pwd && break; done `

pids=""
rc=0

trap 'sigintTrap' 2

sigintTrap() {
	if [ "${pids}X" != "X" ]; then
		kill -9 ${pids}
	fi
	exit 2
}

minikube delete
minikube start

minikube mount ${hereDir}:/issue &
pids="${pids} $!"

minikube dashboard &
pids="${pids} $!"

cat <<EOF | kubectl apply --force -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko-job
spec:
  activeDeadlineSeconds: 1000
  template:
    metadata:
      name: kaniko-job
    spec:
      containers:
      - args:
        - -c
        - |
          /busybox/sh <<'EOF'

          /kaniko/executor \
            --context "/issue/context" \
            --no-push \
            --dockerfile "/issue/context/Dockerfile" \
            --insecure

          EOF
        command: [ "/busybox/sh" ]
        name: kaniko-job
        image: gcr.io/kaniko-project/executor:debug
        volumeMounts:
        - name: issue
          mountPath: /issue
      restartPolicy: OnFailure
      volumes:
      - name: issue
        hostPath:
          path: /issue
EOF

for pid in ${pids}; do
	wait ${pid} || let "rc=1"
done

if [ "${rc}" == "1" ]; then
	exit 1
fi
  • Create a context directory in the issue directory
  • Create the following Dockerfile file under the context directory:
FROM alpine:latest

COPY ./builder.ash /builder.ash

RUN chmod +x /builder.ash && /builder.ash ; rm -rf /builder.ash
  • Create the following builder.ash file under the context directory:
#!/bin/ash

echo "Building..."
  • Give execution permission to the index.bash script: chmod +x issue/index.bash
  • Run the index.bash script: ./issue/index.bash
  • I get the following in the pod:
INFO[0000] Resolved base name alpine:latest to alpine:latest 
INFO[0000] Resolved base name alpine:latest to alpine:latest 
INFO[0000] Retrieving image manifest alpine:latest      
INFO[0001] Retrieving image manifest alpine:latest      
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine:latest      
INFO[0003] Retrieving image manifest alpine:latest      
INFO[0004] Unpacking rootfs as cmd COPY ./builder.ash /builder.ash requires it. 
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_25_19_26_04.767922246: read-only file system

Additional Information

> docker version && minikube version && kubectl version && uname -a

Client: Docker Engine - Community
Version:           19.03.5
API version:       1.40
Go version:        go1.12.12
Git commit:        633a0ea838
Built:             Wed Nov 13 07:29:52 2019
OS/Arch:           linux/amd64
Experimental:      false

Server: Docker Engine - Community
Engine:
 Version:          19.03.5
 API version:      1.40 (minimum version 1.12)
 Go version:       go1.12.12
 Git commit:       633a0ea838
 Built:            Wed Nov 13 07:28:22 2019
 OS/Arch:          linux/amd64
 Experimental:     false
containerd:
 Version:          1.2.10
 GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
 Version:          1.0.0-rc8+dev
 GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
 Version:          0.18.0
 GitCommit:        fec3683
minikube version: v1.6.2
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}
Linux mewtwo-ubuntu 5.3.0-26-generic #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 10
  • Comments: 24 (7 by maintainers)

Most upvoted comments

solution worked for me change version of kaniko: gcr.io/kaniko-project/executor:debug-v0.15.0

Looks like a duplicate of #1001

Bug still present in debug-v0.17.0 - reverting to earlier version fix the build but not ideal.

‘’’ error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/…2020_02_04_10_04_34.525459306: read-only file system ‘’’

Strong recommendation to everyone; use an explicitly tagged release of the kaniko image. Don’t use latest in your critical systems. ~This change has not been released and is only affecting people using latest~ I was incorrect, latest is only built on release. This issue is only affecting people who are using the debug tag which is built on every commit to master. But still, use an explicit version!

@tejal29 I wanted to confirm that v0.17.1 doesn’t have the issue, and if i actively specify --whitelist-var-run=false I can get the same error again - confirming the flag’s actively returning to the other behavior.

Thank you for your work on this! ❤️ 🎉!

I have the same issue (0.17.0):

with --whitelist-var-run=false

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_02_04_12_59_06.412495393: read-only file system

0.16.0 works btw.

@tejal29 What is the solution to this problem? #1001 seems to be related to CloudBuild (which I have no knowledge about). This issue here seems to be related to Kubernetes. AFAIK whitelisting /var/run exposes the service account token in the image, which is a security thread. Not whitelisting breaks the build.

closing this as dupe of #1001