moby: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root767627221: device or resource busy.

Description

Steps to reproduce the issue:

docker run hello-world

Describe the results you received: ~# docker run hello-world Unable to find image ‘hello-world:latest’ locally latest: Pulling from library/hello-world 1b930d010525: Extracting 977B/977B docker: failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root767627221: device or resource busy.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:28:19 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:26:54 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.4.0-146-generic
 Operating System: Ubuntu 16.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 28
 Total Memory: 110.2GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 24 (5 by maintainers)

Links to this issue

newbie asking help upgrading kernel

Most upvoted comments

Linux Kernel upgrade solved it for me: upgraded from 4.19.138 to 5.4.57

+15

Envek on Aug 19, 2020

Faced the same problem during docker pull command.

$ docker pull node
Using default tag: latest
latest: Pulling from library/node
419e7ae5bb1e: Already exists 
848839e0cd3b: Already exists 
de30e8b35015: Already exists 
258fdea6ea48: Already exists 
ddb75eb7f1e9: Already exists 
7ec8a0667334: Already exists 
c6d8dab5779d: Extracting [==================================================>]  34.68MB/34.68MB
d3ae9be01b42: Download complete 
38ff96b6f833: Download complete 
failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root223330811: device or resource busy

Next attempts failing immediately:

$ docker pull node
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

The only info I can found in journalctl is:

Aug 19 12:22:43 Envek-NUC dockerd[1353]: time="2020-08-19T12:22:43.526767832+03:00" level=info msg="Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): Error cleaning up after pivot: remove /.pivot_root277226559: device or resource busy"

systemctl says that docker is working:

$ sudo systemctl status docker                                              
● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2020-08-19 12:21:42 MSK; 17min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1353 (dockerd)
      Tasks: 30
     Memory: 227.6M
     CGroup: /system.slice/docker.service
             └─1353 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Aug 19 12:21:41 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:41.847801513+03:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Aug 19 12:21:41 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:41.847904990+03:00" level=info msg="Loading containers: start."
Aug 19 12:21:42 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:42.073824102+03:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set >
Aug 19 12:21:42 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:42.105989916+03:00" level=info msg="Loading containers: done."
Aug 19 12:21:42 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:42.252641747+03:00" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CON>
Aug 19 12:21:42 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:42.252901832+03:00" level=info msg="Docker daemon" commit=48a66213fe graphdriver(s)=overlay2 version=19.03.12-ce
Aug 19 12:21:42 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:42.253573843+03:00" level=info msg="Daemon has completed initialization"
Aug 19 12:21:42 Envek-NUC dockerd[1353]: time="2020-08-19T12:21:42.282312388+03:00" level=info msg="API listen on /run/docker.sock"
Aug 19 12:21:42 Envek-NUC systemd[1]: Started Docker Application Container Engine.
Aug 19 12:22:43 Envek-NUC dockerd[1353]: time="2020-08-19T12:22:43.526767832+03:00" level=info msg="Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status>

Restarting docker service doesn’t help. Only system reboot helps.

And in general system starts to behave weirdly: sudo is asking for password after ≈10 seconds delay. So may be it is not docker issue and something suddenly broke in my system, but what? Disk space seems to be ok.

docker version

Client:
 Version:           19.03.12-ce
 API version:       1.40
 Go version:        go1.14.5
 Git commit:        48a66213fe
 Built:             Sat Jul 18 01:33:21 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.12-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.14.5
  Git commit:       48a66213fe
  Built:            Sat Jul 18 01:32:59 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.4.m
  GitCommit:        d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker info

Client:
 Debug Mode: false

Server:
 Containers: 24
  Running: 0
  Paused: 0
  Stopped: 24
 Images: 502
 Server Version: 19.03.12-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.138-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.29GiB
 Name: Envek-NUC
 ID: YOSA:APWU:QHEK:KOTI:KHVS:XBSY:TEPV:EFKI:YOVR:2BGD:7QV2:7P4S
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  localhost:32000
  127.0.0.0/8
 Live Restore Enabled: false

OS: Manjaro Linux. Docker installed via package manager.

+14

Envek on Aug 19, 2020

Yes upgrading kernel worked for me too. Just a quick reminder how to upgrade a kernel: https://wiki.manjaro.org/index.php/Manjaro_Kernels

sirex on Aug 20, 2020

Bug was introduced into Manjaro linux419 (https://gitlab.manjaro.org/packages/core/linux419) at commit 98da16f.

Specifically, the problem is proc_mounts.patch which was added then.

Last good release was 4.19.133 (if it was released - cannot be sure) or 4.19.132.

You can build the latest kernel without proc_mounts.patch. Three modifications are required in PKGBUILD: the call to patch; the source; the checksum.

I have tested this on 4.19.152. It does fix the problem.

Applying proc_mounts.patch to Manjaro 5.4 recreates the exact same problem.

Applying proc_mounts.patch to ArchLinux 5.4 does not create the problem.

I think the aufs patches are tripping over themselves. But I’m not yet sure.

Furthermore, I believe the problem would affect containers in general and not just docker.

I have thoughts about the true root cause. But they’re not much further beyond speculation. So I decided to post this much at the moment to help anyone stuck.

joseph-r-hamilton on Nov 23, 2020

The same problem happened to me, also.

And /etc/mtab is broken.

yssource on Oct 26, 2020

I faced exactly the same issue on Manjaro with 4.19.143-1-MANJARO x86_64 after docker updated to 19.03.12-ce Upgrading kernel to 5.4.62-1-MANJARO x86_64 solved problem.

Minilfat on Sep 10, 2020

Created new ticket in Arch bug tracker: FS#67639

Envek on Aug 19, 2020