podman: error allocating lock for new pod: no space left on device
/kind bug
Description
I rarely change my container setup, but if I go there after some weeks/couple months to change something, I can’t spawn new containers due:
Steps to reproduce the issue:
- Run podman, any container. here is simple sample:
sudo podman run --name sftp --rm -p 2222:22/tcp -h sftp --memory=128M atmoz/sftp:latest
Error: error allocating lock for new container: no space left on device
Describe the results you received:
It always fails. I updated the podman to latest 1.5.1-3, and it still fails the same. Note, there were containers running over upgrade.
I can’t see what it tries to open:
sudo strace -fe open podman run --name sftp --rm -p 2222:22/tcp -h sftp --memory=128M atmoz/sftp:latest --log-level=debug
strace: Process 17668 attached
strace: Process 17669 attached
strace: Process 17670 attached
strace: Process 17671 attached
strace: Process 17672 attached
strace: Process 17673 attached
strace: Process 17674 attached
strace: Process 17675 attached
strace: Process 17676 attached
[pid 17676] +++ exited with 0 +++
[pid 17671] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=17676, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
Trying to pull docker.io/atmoz/sftp:latest...
strace: Process 17677 attached
strace: Process 17678 attached
strace: Process 17679 attached
strace: Process 17680 attached
Getting image source signatures
strace: Process 17701 attached
strace: Process 17704 attached
strace: Process 17705 attached
Copying blob 017cdea0acaa done
Copying blob 03cce7e7c0ee done
Copying blob b2f8f2e93ab3 done
Copying blob 54f7e8ac135a done
Copying blob 0fbd7701cad1 done
Copying config 6345f82053 done
Writing manifest to image destination
Storing signatures
Error: error allocating lock for new container: no space left on device
[pid 17704] +++ exited with 125 +++
[pid 17705] +++ exited with 125 +++
[pid 17701] +++ exited with 125 +++
[pid 17680] +++ exited with 125 +++
[pid 17679] +++ exited with 125 +++
[pid 17678] +++ exited with 125 +++
[pid 17677] +++ exited with 125 +++
[pid 17675] +++ exited with 125 +++
[pid 17674] +++ exited with 125 +++
[pid 17673] +++ exited with 125 +++
[pid 17672] +++ exited with 125 +++
[pid 17671] +++ exited with 125 +++
[pid 17670] +++ exited with 125 +++
[pid 17669] +++ exited with 125 +++
[pid 17668] +++ exited with 125 +++
+++ exited with 125 +++
I have plenty of space both in bytes as well as in inodes:
$ sudo df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.8G 0 3.8G 0% /dev
tmpfs 3.8G 168K 3.8G 1% /dev/shm
tmpfs 3.8G 17M 3.8G 1% /run
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/mapper/fedora-root 49G 33G 15G 70% /
tmpfs 3.8G 72K 3.8G 1% /tmp
/dev/sda1 976M 205M 705M 23% /boot
/dev/mapper/fedora-home 177G 24G 144G 14% /home
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/61ecc54cf1c77e8c2877b88b61838356c662496147
43e78e2e37d334dfab1123/userdata/shm
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/00b91bda33dcee9a0fc8502b7e0eb28ee06de5ca50524fed28a37
85c75c62c9d/merged
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/fb406050a05b6db12fceecafc59e27cde30be5b2108c7be563d5a
6efed97b67e/merged
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/cb450a422b3da1f655aef09144af5c426a2e11a4f0
33651b725cf5e8186c1360/userdata/shm
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/b3f61b802a3041d283a9c315d9bfb17d517873f382d9bd87ccc4a
e817bbc98bc/merged
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/0aa8bd7512a04f5c03aee311585b5048988a2a9bf5
71d6b90a8d51dc75e54f33/userdata/shm
tmpfs 768M 16K 768M 1% /run/user/1017
tmpfs 768M 20K 768M 1% /run/user/42
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/f21ff8e7c827dde1bde9f44cf4a304a8c65808cd54bc8618ad07f
12542dbb1c5/merged
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/2e20e07861960b51dd83534369e04d7b4a16a70f57
c43054bb67980b66662eb7/userdata/shm
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/079108eedaffc503ba3e9a792f4a10caedde8e776a
b01f79363e443d148ce901/userdata/shm
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/2eaf7978d284e70422ea28d1f49933be05000fb428ad808891b33
a518680c44c/merged
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/860832aff0dadc395990f5a4e178284a8ad25ea049
3e08e78a00acea840460d0/userdata/shm
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/999bcb15197d19f9eb3d2330b7afb91f09ba08af743ebffa536ed
4a85bdddea7/merged
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/b19e44a179757de5962d302cb0600f53dc1c787f8b
507f37ab111c40c6480744/userdata/shm
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/07932579f56d5af5753f370ea96235129ce2c92edb00e071154d5
3c68e6dde15/merged
shm 63M 0 63M 0% /var/lib/containers/storage/overlay-containers/7cd90271f81f6ee84486c5d0fae27fade8b9e8f241
6129ae4a94323f3fb0a188/userdata/shm
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/a8c46521b6d0cfbbf6252931cd5a36798f638b532cc84a64fa3bf
a85b115cf2a/merged
shm 63M 6.4M 57M 11% /var/lib/containers/storage/overlay-containers/4be11f7879f384bbb50a228b07b48e9460db7818cf
b370e8671e23f5c4b83576/userdata/shm
overlay 49G 33G 15G 70% /var/lib/containers/storage/overlay/2b8b2ffae7f9f4f58e262df9601c738d4298511a3c4241839ea71
482436039ae/merged
$ sudo df -hi
Filesystem Inodes IUsed IFree IUse% Mounted on
devtmpfs 957K 550 956K 1% /dev
tmpfs 960K 3 960K 1% /dev/shm
tmpfs 960K 4.9K 956K 1% /run
tmpfs 960K 17 960K 1% /sys/fs/cgroup
/dev/mapper/fedora-root 3.2M 581K 2.6M 19% /
tmpfs 960K 22 960K 1% /tmp
/dev/sda1 64K 444 64K 1% /boot
/dev/mapper/fedora-home 12M 173K 12M 2% /home
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/61ecc54cf1c77e8c2877b88b61838356c662496$
4743e78e2e37d334dfab1123/userdata/shm
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/00b91bda33dcee9a0fc8502b7e0eb28ee06de5ca50524fed28$
3785c75c62c9d/merged
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/fb406050a05b6db12fceecafc59e27cde30be5b2108c7be563$
5a6efed97b67e/merged
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/cb450a422b3da1f655aef09144af5c426a2e11a$
f033651b725cf5e8186c1360/userdata/shm
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/b3f61b802a3041d283a9c315d9bfb17d517873f382d9bd87cc$
4ae817bbc98bc/merged
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/0aa8bd7512a04f5c03aee311585b5048988a2a9$
f571d6b90a8d51dc75e54f33/userdata/shm
tmpfs 960K 38 960K 1% /run/user/1017
tmpfs 960K 20 960K 1% /run/user/42
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/f21ff8e7c827dde1bde9f44cf4a304a8c65808cd54bc8618ad$
7f12542dbb1c5/merged
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/2e20e07861960b51dd83534369e04d7b4a16a70$
57c43054bb67980b66662eb7/userdata/shm
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/079108eedaffc503ba3e9a792f4a10caedde8e7$
6ab01f79363e443d148ce901/userdata/shm
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/2eaf7978d284e70422ea28d1f49933be05000fb428ad808891$
33a518680c44c/merged
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/860832aff0dadc395990f5a4e178284a8ad25ea$
493e08e78a00acea840460d0/userdata/shm
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/999bcb15197d19f9eb3d2330b7afb91f09ba08af743ebffa53$
ed4a85bdddea7/merged
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/b19e44a179757de5962d302cb0600f53dc1c787$
8b507f37ab111c40c6480744/userdata/shm
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/07932579f56d5af5753f370ea96235129ce2c92edb00e07115$
d53c68e6dde15/merged
shm 960K 1 960K 1% /var/lib/containers/storage/overlay-containers/7cd90271f81f6ee84486c5d0fae27fade8b9e8f$
416129ae4a94323f3fb0a188/userdata/shm
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/a8c46521b6d0cfbbf6252931cd5a36798f638b532cc84a64fa$
bfa85b115cf2a/merged
shm 960K 10 960K 1% /var/lib/containers/storage/overlay-containers/4be11f7879f384bbb50a228b07b48e9460db781$
cfb370e8671e23f5c4b83576/userdata/shm
overlay 3.2M 581K 2.6M 19% /var/lib/containers/storage/overlay/2b8b2ffae7f9f4f58e262df9601c738d4298511a3c4241839e$
71482436039ae/merged
Describe the results you expected:
Container to start.
Additional information you deem important (e.g. issue happens only occasionally):
I have some containers mounted to NFS share, if that could make a difference.
Output of podman version:
$ podman version
Version: 1.5.1
RemoteAPI Version: 1
Go Version: go1.11.12
OS/Arch: linux/amd64
Output of podman info --debug:
debug:
compiler: gc
git commit: ""
go version: go1.11.12
podman version: 1.5.1
host:
BuildahVersion: 1.10.1
Conmon:
package: podman-1.5.1-3.fc29.x86_64
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.0, commit: fa55639b725e7626b28dbd43de8e9546f7411226-dirty'
Distribution:
distribution: fedora
version: "29"
MemFree: 636534784
MemTotal: 8052482048
OCIRuntime:
package: runc-1.0.0-93.dev.gitb9b6cc6.fc29.x86_64
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc8+dev
commit: 82f4855a8421018c9f4d74fbcf2da7f8ad1e11fa
spec: 1.0.1-dev
SwapFree: 8054894592
SwapTotal: 8199860224
arch: amd64
cpus: 4
eventlogger: journald
hostname: ohuska.localdomain
kernel: 5.1.8-200.fc29.x86_64
os: linux
rootless: false
uptime: 1142h 55m 33.37s (Approximately 47.58 days)
registries:
blocked: null
insecure: null
search:
- docker.io
- registry.fedoraproject.org
- quay.io
- registry.access.redhat.com
- registry.centos.org
store:
ConfigFile: /etc/containers/storage.conf
ContainerStore:
number: 11
GraphDriverName: overlay
GraphOptions:
- overlay.mountopt=nodev
GraphRoot: /var/lib/containers/storage
GraphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
ImageStore:
number: 13
RunRoot: /var/run/containers/storage
VolumePath: /var/lib/containers/storage/volumes
Additional environment details (AWS, VirtualBox, physical, etc.):
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 1
- Comments: 48 (21 by maintainers)
I had this issue, and it turned out that, according to
podman system df, I had 2047 local volumes. I didpodman volume prune, followed by apodman system renumber` for good luck, because I’m basically cargo-culting here, and my container worked after that.I just did a test starting two containers and then stopping them using the systemd service file I posted above. I then did a
podman volume pruneand it deleted two volumes. This means that these two volumes were left as garbage when the containers stopped. (I also did the test manually starting -> stopping -> removing a container and the result was again leftover volumes)Thanks a lot for sharing!
Can you try without
--rm \in the ExecStart?