moby: Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument

Description

bash-4.3# docker logs -f kontena-agent
Error response from daemon: readlink /var/lib/docker/overlay2/l: invalid argument

Steps to reproduce the issue:

have overlay2 and extfs4 mounted in /var/lib/docker
run containers
reboot the machine by power cycling it hard enough

Output of docker version:

Docker version 17.09.0-ce, build afdb6d4

Output of docker info:

Docker version 17.09.0-ce, build afdb6d4
bash-4.3# docker info
Containers: 9
 Running: 2
 Paused: 0
 Stopped: 7
Images: 10
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: v0.13.2 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 4.13.16-coreos-r2
Operating System: Container Linux by CoreOS 1576.4.0 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.863GiB
Name: pannu
ID: ZJ6X:DLDI:4MHS:3DIG:WQUY:SFBI:SCUC:7URO:CYDC:4DR6:UI56:XGCJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 region=virtualbox
 ephemeral=yes
Experimental: false
Insecure Registries:
 10.81.0.0/16
 127.0.0.0/8
Live Restore Enabled: false

$ cat /etc/lsb-release
DISTRIB_ID="Container Linux by CoreOS"
DISTRIB_RELEASE=1576.4.0
DISTRIB_CODENAME="Ladybug"
DISTRIB_DESCRIPTION="Container Linux by CoreOS 1576.4.0 (Ladybug)"

$ uname
Linux pannu 4.13.16-coreos-r2 #1 SMP Wed Dec 6 04:27:34 UTC 2017 x86_64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz GenuineIntel GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.):

Virtualbox

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 2
Comments: 29 (9 by maintainers)

Most upvoted comments

So you’re saying there is no acceptable solution at the moment?

DaDummy on Sep 6, 2019

Clean/Purge data on Docker app desktop

cdecarlos on May 12, 2021

Skimming the overlay2 code, there’s a direct call to readlink when it’s looking for lower directories

The errors in this issue could happen if the lower file were corrupt, or if the write to it were short. That file being truncated/not fully written would lead to the errors described here I think.

@discordianfish If you check the contents of the lower files in /var/lib/docker/overlay2/$id/lower, I think you might find one that ends with a : character rather than with a valid path. Running head -n9999 /var/lib/docker/overlay2/*/lower as root should display each “lower” file and its contents. Can you either paste that output somewhere or verify that one of them is erroneous?

Since docker isn’t doing an atomic write+move operation to update those files, there’s numerous ways in which they could end up truncated in such a way. If these errors are the result of corrupt lower files, switching to an atomic update of those files should resolve the issue.

@thaJeztah No worries about the ping; I’m still happy to hack on container-stuff regardless. For CoreOS-related issues, @bgilbert should be able to help with them.

euank on Jun 13, 2019

I have traced get more new information. The problem layer link and lower file’s content is empty:

committed  diff  link  lower  work

cat link
cat lower

This will cause Driver.getLower in daemon/graphdriver/overlay2/overlay.go get parentLink with “”, and lowers is [l]

lowers := []string{path.Join(linkDir, string(parentLink))}

And parentLower is “”, which lead parentLowers get []

parentLowers := strings.Split(string(parentLower), ":")

And finally, lowers will get [l, ]. Driver.getLower function will return l:

lowers = append(lowers, parentLowers...)
...
return strings.Join(lowers, ":"), nil

zvier on Oct 17, 2021

Skimming the overlay2 code, there’s a direct call to readlink when it’s looking for lower directories

The errors in this issue could happen if the lower file were corrupt, or if the write to it were short. That file being truncated/not fully written would lead to the errors described here I think.

@discordianfish If you check the contents of the lower files in /var/lib/docker/overlay2/$id/lower, I think you might find one that ends with a : character rather than with a valid path. Running head -n9999 /var/lib/docker/overlay2/*/lower as root should display each “lower” file and its contents. Can you either paste that output somewhere or verify that one of them is erroneous?

Since docker isn’t doing an atomic write+move operation to update those files, there’s numerous ways in which they could end up truncated in such a way. If these errors are the result of corrupt lower files, switching to an atomic update of those files should resolve the issue.

@thaJeztah No worries about the ping; I’m still happy to hack on container-stuff regardless. For CoreOS-related issues, @bgilbert should be able to help with them.

I also meet this problem in the latest version docker 20.10. And rmi image and pull again or restart dockerd can not slove.

zvier on Oct 17, 2021

Hitting this regularly with:

Docker: 18.3.1
OS: Ubuntu 16.0.4 LTS
Kernel: 4.4.0-109-generic

Duplicate issues elsewhere:

calder on Sep 28, 2018

I saw this error today on a dind 17.11.0-ce on Linuxkit, on XFS. I wiped /var/lib/docker and the issue went away. Not sure the cause offhand.

eyz on Dec 11, 2017