containerd: No space on device error using gitlab runner
Description I am using docker with gitlab runner and have encountered a problem where contanerd cannot create a container. The error output from gitlab runner is as follows:
ERROR: Job failed (system failure): Error response from daemon: mkdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/f947041d64b7df21483d9dc1f6daba0f2a4822c802b98e58e3550c6bc75599fa: no space left on device: unknown (executor_docker.go:834:0s)
People wrote that such an error can occur when using the ex4 file system, which is used on my system. In order to solve this problem, a separate place was allocated for temporary containerd files using the xfs file system. I registered the paths to this directory for containerd (in the config.toml file) and for docker (in the service file) restarted services, but the error didn’t disappear, although I can observe that containerd works with the directory that was written in the config. Problem also didn’t disappear after clearing system from all containerd files and reinstalling and recongiguring it. My containerd config.toml file looks like this:
disabled_plugins = ["cri"]
root = "/docker-disk/var/lib/containerd"
state = "/docker-disk/run/containerd"
subreaper = true
oom_score = 0
[grpc]
address = "/docker-disk/run/containerd/containerd.sock"
uid = 0
gid = 0
[debug]
address = "/docker-disk/run/containerd/debug.sock"
uid = 0
gid = 0
level = "info"
Containerd version:
containerd containerd.io 1.2.13 7ad184331fa3e55e52b890ea95e65ba581ae3429
Docker version:
Version: 19.03.9
Any other relevant information:
OS
CentOS Linux release 7.7.1908
Kernel version
3.10.0-1062.12.1.el7.x86_64
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 17 (3 by maintainers)
Hi @luckyl3, can you provide a link or explanation how you solved the GitLab problem? Thanks and best!
I solved the issue by deleting old images
docker image prune -a --filter "until=24h"
explanation please, same problem here
I solved my problem, it was not containerd problem, but it was gitlab problem, as it turned out. Thanks for trying to help me. I close the issue.
This happens sometimes on gitlab, it’s not container problem. This error was resolved by itself on the next day for me