moby: Failed to restart daemon after 24.0.2 -> 24.0.3 upgrade
Description
After update on all hosts (20+, all that I’m running) docker daemon is failed.
error is:
Jul 07 03:03:17 server systemd[1]: Starting Docker Application Container Engine...
Jul 07 03:03:17 server dockerd[52024]: time="2023-07-07T03:03:17.076517762Z" level=info msg="Starting up"
Jul 07 03:03:17 server dockerd[52024]: time="2023-07-07T03:03:17.086137356Z" level=info msg="[graphdriver] using prior storage driver: fuse-overlayfs"
Jul 07 03:03:17 server dockerd[52024]: time="2023-07-07T03:03:17.103391077Z" level=info msg="Loading containers: start."
Jul 07 03:03:17 server dockerd[52024]: time="2023-07-07T03:03:17.411734779Z" level=info msg="there are running containers, updated network configuration will not take affect"
Jul 07 03:03:17 server dockerd[52024]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 07 03:03:17 server dockerd[52024]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x564e9be4c7c0]
Jul 07 03:03:17 server dockerd[52024]: goroutine 607 [running]:
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).prepareMountPoints(0xc00054b7b8?, 0xc0007c2500)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/mounts.go:24 +0x1c0
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).restore.func5(0xc0007c2500, 0x0?)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:552 +0x271
Jul 07 03:03:17 server dockerd[52024]: created by github.com/docker/docker/daemon.(*Daemon).restore
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:530 +0x8d8
Jul 07 03:03:17 server dockerd[52024]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 07 03:03:17 server dockerd[52024]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x564e9be4c7c0]
Jul 07 03:03:17 server dockerd[52024]: goroutine 601 [running]:
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).prepareMountPoints(0xc00054b7b8?, 0xc000cb2000)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/mounts.go:24 +0x1c0
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).restore.func5(0xc000cb2000, 0x0?)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:552 +0x271
Jul 07 03:03:17 server dockerd[52024]: created by github.com/docker/docker/daemon.(*Daemon).restore
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:530 +0x8d8
Jul 07 03:03:17 server dockerd[52024]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 07 03:03:17 server dockerd[52024]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x564e9be4c7c0]
Jul 07 03:03:17 server dockerd[52024]: goroutine 603 [running]:
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).prepareMountPoints(0xc00054b7b8?, 0xc000187680)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/mounts.go:24 +0x1c0
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).restore.func5(0xc000187680, 0x0?)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:552 +0x271
Jul 07 03:03:17 server dockerd[52024]: created by github.com/docker/docker/daemon.(*Daemon).restore
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:530 +0x8d8
Jul 07 03:03:17 server dockerd[52024]: panic: runtime error: invalid memory address or nil pointer dereference
Jul 07 03:03:17 server dockerd[52024]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x564e9be4c7c0]
Jul 07 03:03:17 server dockerd[52024]: goroutine 600 [running]:
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).prepareMountPoints(0xc00054b7b8?, 0xc00063e500)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/mounts.go:24 +0x1c0
Jul 07 03:03:17 server dockerd[52024]: github.com/docker/docker/daemon.(*Daemon).restore.func5(0xc00063e500, 0xc00005e600?)
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:552 +0x271
Jul 07 03:03:17 server dockerd[52024]: created by github.com/docker/docker/daemon.(*Daemon).restore
Jul 07 03:03:17 server dockerd[52024]: /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:530 +0x8d8
I’m running daemon with config:
{
"live-restore": true
}
any chance to manage restart daemon without restarting containers?
Reproduce
systemctl restart docker
Expected behavior
systemd service to start
docker version
unable to provide, as service is down
docker info
unable to provide, as service is down
Additional Info
cant provide docker info as service do not start, I have containers running on servers.
If i reboot server (i.e. containers get stopped, docker starts without issues)
About this issue
- Original URL
- State: closed
- Created a year ago
- Reactions: 7
- Comments: 15 (8 by maintainers)
Issue fixed with 24.0.4 update.
the panic is caused by scripts updating running container network configs. I worked around this:
sudo apt-get install docker-ce="5:24.0.2-1~ubuntu.22.04~jammy"docker stop $(docker ps -a -q)docker start $(docker ps -a -q)However, restarting the docker service causes the issue again, so I recommend staying on the previous version.
It’s empty for bind-mounts etc, which don’t need restoring
thank you @tomsiewert @calvinbui
If anyone found solution to apply this update without restarting containers, please share.