dokku: Old commit's containers do not ever shut down
Description of problem
Old commit’s containers do not ever shut down
How reproducible
Occasionally (maybe 1 in 10-15 deploys)
Steps to Reproduce
- Deploy new version of app with multiple process types/containers
- Notice that jobs embedded in old containers are still executing
- Check
sudo docker psand see old containers still running alongside latest commit’s containers
Actual Results
Containers of old application commits remain running after new commit/deploy
Expected Results
Containers of old application commits are shut down ~1 minute after new commit/deploy
Environment Information
dokku report APP_NAME output
michaelyork@apps:~$ dokku report REDACTED_APP_NAME
-----> uname: Linux apps 4.4.0-128-generic #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 9998 2255 2704 92 5037 7131
Swap: 974 0 974
-----> docker version:
Client:
Version: 18.05.0-ce
API version: 1.37
Go version: go1.9.5
Git commit: f150324
Built: Wed May 9 22:16:25 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.05.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: f150324
Built: Wed May 9 22:14:32 2018
OS/Arch: linux/amd64
Experimental: false
-----> docker daemon info:
Containers: 53
Running: 37
Paused: 0
Stopped: 16
Images: 120
Server Version: 18.05.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
WARNING: No swap limit support
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-128-generic
Operating System: Ubuntu 16.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 9.764GiB
Name: apps
ID: RF4A:Y5ZM:44CL:KAQM:LA2T:TZEM:EBFE:CIE6:6WKR:FNSU:PIGB:XMCD
Docker Root Dir: /var/lib/docker
Debug Mode (client): true
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-----> sigil version: 0.4.0
-----> herokuish version:
herokuish: 0.4.0
buildpacks:
heroku-buildpack-multi v1.0.0
heroku-buildpack-ruby v174
heroku-buildpack-nodejs v121
heroku-buildpack-clojure v78
heroku-buildpack-python v124
heroku-buildpack-java v59
heroku-buildpack-gradle v25
heroku-buildpack-scala v79
heroku-buildpack-play v26
heroku-buildpack-php v132
heroku-buildpack-go v85
buildpack-nginx v9
-----> dokku version: 0.12.5
-----> dokku plugins:
plugn: 0.3.0
00_dokku-standard 0.12.5 enabled dokku core standard plugin
20_events 0.12.5 enabled dokku core events logging plugin
apps 0.12.5 enabled dokku core apps plugin
build-env 0.12.5 enabled dokku core build-env plugin
certs 0.12.5 enabled dokku core certificate management plugin
checks 0.12.5 enabled dokku core checks plugin
common 0.12.5 enabled dokku core common plugin
config 0.12.5 enabled dokku core config plugin
docker-options 0.12.5 enabled dokku core docker-options plugin
domains 0.12.5 enabled dokku core domains plugin
enter 0.12.5 enabled dokku core enter plugin
git 0.12.5 enabled dokku core git plugin
letsencrypt 0.8.8 enabled Automated installation of let's encrypt TLS certificates
logs 0.12.5 enabled dokku core logs plugin
memcached 1.2.7 enabled dokku memcached service plugin
network 0.12.5 enabled dokku core network plugin
nginx-vhosts 0.12.5 enabled dokku core nginx-vhosts plugin
plugin 0.12.5 enabled dokku core plugin plugin
postgres 1.2.7 enabled dokku postgres service plugin
proxy 0.12.5 enabled dokku core proxy plugin
ps 0.12.5 enabled dokku core ps plugin
redis 1.2.7 enabled dokku redis service plugin
repo 0.12.5 enabled dokku core repo plugin
scheduler-docker-local 0.12.5 enabled dokku core scheduler-docker-local plugin
shell 0.12.5 enabled dokku core shell plugin
ssh-keys 0.12.5 enabled dokku core ssh-keys plugin
storage 0.12.5 enabled dokku core storage plugin
tags 0.12.5 enabled dokku core tags plugin
tar 0.12.5 enabled dokku core tar plugin
=====> REDACTED_APP_NAME app information
App dir: /home/dokku/REDACTED_APP_NAME
Git sha: 4a7cdf3
Deploy source: git
Locked: false
=====> REDACTED_APP_NAME ssl information
Ssl dir: /home/dokku/REDACTED_APP_NAME/tls
Ssl enabled: true
Ssl hostnames: REDACTED_APP_NAME.com
Ssl expires at: Sep 24 00:43:50 2018 GMT
Ssl issuer: C=US, O=Lets Encrypt, CN=Lets Encrypt Authority X3
Ssl starts at: Jun 26 00:43:50 2018 GMT
Ssl subject: CN=REDACTED_APP_NAME.com
Ssl verified: self signed
=====> REDACTED_APP_NAME checks information
Checks disabled list: none
Checks skipped list: none
=====> REDACTED_APP_NAME docker options information
Docker options build: --link dokku.redis.REDACTED_APP_NAME:dokku-redis-REDACTED_APP_NAME
Docker options deploy: --link dokku.redis.REDACTED_APP_NAME:dokku-redis-REDACTED_APP_NAME --restart=on-failure:10
Docker options run: --link dokku.redis.REDACTED_APP_NAME:dokku-redis-REDACTED_APP_NAME
=====> REDACTED_APP_NAME domains information
Domains app enabled: true
Domains app vhosts: REDACTED_APP_NAME.com
Domains global enabled: true
Domains global vhosts: REDACTED_URL
=====> REDACTED_APP_NAME git information
Git rev env var: GIT_REV
Git deploy branch: master
Git global deploy branch: master
=====> REDACTED_APP_NAME network information
Network bind all interfaces: false
Network listeners: 172.17.0.28:5000
=====> REDACTED_APP_NAME proxy information
Proxy enabled: true
Proxy type: nginx
Proxy port map: http:80:5000 https:443:5000
=====> REDACTED_APP_NAME ps information
Processes: 8
Deployed: true
Running: true
Restore: true
Restart policy: on-failure:10
Status user_vehicle_queuer.1: running (CID: 4e575fa82566)
Status vehicle_data_queuer.1: running (CID: ad1d4e5823e7)
Status web.1: running (CID: ea0af908934a)
Status worker.1: running (CID: bc3a8fe63d5b)
Status worker.2: running (CID: fcba0b6e9d43)
Status worker.3: running (CID: bb9d26bf361d)
Status worker.4: running (CID: 7765f140c777)
Status worker.5: running (CID: 71fcc68d4ca9)
=====> REDACTED_APP_NAME storage information
Storage build mounts:
Storage deploy mounts:
Storage run mounts:
How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:
bootstrap script on physical/Ubuntu 16
Additional information
docker inspect CONTAINER_ID(if applicable): (BEWARE:docker inspectwill print environment variables for some commands, be sure you’re not exposing any sensitive information when posting issues. You may replace these values with XXXXXXX):
This is from one of the containers that should have been shut down over 4 days ago.
michaelyork@apps:~$ sudo docker inspect 9396b28b8636
[
{
"Id": "9396b28b86365ca73039e273d475d82abcb043f48c168c343208bb0b98aff140",
"Created": "2018-07-02T05:21:25.6958793Z",
"Path": "/start",
"Args": [
"worker"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 29228,
"ExitCode": 0,
"Error": "",
"StartedAt": "2018-07-02T05:21:26.102475281Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:4e7c1e80b184d64eab15177278d67c050b21a6df6256bad5cfe5de12c590b5f3",
"ResolvConfPath": "/var/lib/docker/containers/9396b28b86365ca73039e273d475d82abcb043f48c168c343208bb0b98aff140/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/9396b28b86365ca73039e273d475d82abcb043f48c168c343208bb0b98aff140/hostname",
"HostsPath": "/var/lib/docker/containers/9396b28b86365ca73039e273d475d82abcb043f48c168c343208bb0b98aff140/hosts",
"LogPath": "/var/lib/docker/containers/9396b28b86365ca73039e273d475d82abcb043f48c168c343208bb0b98aff140/9396b28b86365ca73039e273d475d82abcb043f48c168c343208bb0b98aff140-json.log",
"Name": "/REDACTED_APP_NAME.worker.5.1530510758",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {},
"RestartPolicy": {
"Name": "on-failure",
"MaximumRetryCount": 10
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "shareable",
"Cgroup": "",
"Links": [
"/dokku.redis.REDACTED_APP_NAME:/REDACTED_APP_NAME.worker.5.1530510758/dokku-redis-REDACTED_APP_NAME"
],
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/ad1e8e9dd8b4787384278088f9ecdb319659aecea20f0304fce69d9547b630d4-init/diff:/var/lib/docker/overlay2/100e6aa11ba37d983425ecd47bc16e08609fcda399a3afe310d6b8aa3319bc6a/diff:/var/lib/docker/overlay2/fd463978306d710331f1e84218c759decf79b427a43ede37fbba21bc4b600df5/diff:/var/lib/docker/overlay2/2b5f5f7b7797ee87cbbe31f11cfed13eba976f7d63f5ab6a40c08e16fd64c1c8/diff:/var/lib/docker/overlay2/436b219d9910cd48983a8698c59baace800cc721081abec5999349140b775836/diff:/var/lib/docker/overlay2/8bebe7dc2f0a76d26b3d881b1da8fc06d0f66e8febc5f3fc28ba57cb1fdb0f61/diff:/var/lib/docker/overlay2/224d7991ebc5f83ec36908dbd7ae73cf771a4ad60a47709da1e5f7c4f8c38416/diff:/var/lib/docker/overlay2/1fc2578c7f691514f7aa964a746eb17d5ec797b01ae591a0c76fcfe6cf79eb5f/diff:/var/lib/docker/overlay2/f0f2706db1068a55fba4528b9796d5544a635c8035e21f9323ea5339e95d87ce/diff:/var/lib/docker/overlay2/0a3e1e5f35edcaec7203a536c269aeb077773f0838d6b2ca2d38f5c404f9170d/diff:/var/lib/docker/overlay2/dc9300e8679b50a89a67f8270bbb9dc458e8b518ce141608a719f6a54cf2aa81/diff:/var/lib/docker/overlay2/a32e50538de5463d806fc55b40b0ae5c29d11c0489a440774791e0253cb23cff/diff:/var/lib/docker/overlay2/1cc6a34a03b8e841bb6e76210d8a517436c189f13ed4ad08ab27f18474f9793a/diff:/var/lib/docker/overlay2/9887e1583e0466e8d71b07fec67a7c5ef2382e0a4f4279e8520d4cc87ff0257c/diff:/var/lib/docker/overlay2/1efe336fa59415a243798627be1b7c7eb5e3b45a92642c7aadcc15d4e3bb713c/diff:/var/lib/docker/overlay2/dddbb75119a5f896462bb3867d106292238dcc8bd6eecf82c0cb85c985c06573/diff:/var/lib/docker/overlay2/001a523a9b300352226677079af7e3040c16401e5347713a0db97194f932987f/diff:/var/lib/docker/overlay2/428a3cb22a4e5cba9b7d5001e85deeebdd6dbd6d397ec9d52effff5a690acf4a/diff:/var/lib/docker/overlay2/540c49ac522576cda40365718999a91f108a6896094f105d2200a219208b06e3/diff:/var/lib/docker/overlay2/9311560a9cdd8ce8bf0c32aca9d287b5f37dcfb08128a143ef743dfa8e5f4274/diff:/var/lib/docker/overlay2/c42b5026dd5deedd12ff95302274bfef0292d0ba55e1aef61285837410cf85fa/diff",
"MergedDir": "/var/lib/docker/overlay2/ad1e8e9dd8b4787384278088f9ecdb319659aecea20f0304fce69d9547b630d4/merged",
"UpperDir": "/var/lib/docker/overlay2/ad1e8e9dd8b4787384278088f9ecdb319659aecea20f0304fce69d9547b630d4/diff",
"WorkDir": "/var/lib/docker/overlay2/ad1e8e9dd8b4787384278088f9ecdb319659aecea20f0304fce69d9547b630d4/work"
},
"Name": "overlay2"
},
"Mounts": [],
"Config": {
"Hostname": "9396b28b8636",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"USER=herokuishuser",
"DYNO=worker.5",
"CACHE_PATH=/cache",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"DEBIAN_FRONTEND=noninteractive"
],
"Cmd": [
"/start",
"worker"
],
"Image": "dokku/REDACTED_APP_NAME:latest",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": {
"dokku": ""
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "c7f74ac1d35282940f4e8782d0dc829602b8f03d9cd73f7c7fe58eaefe84735d",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {},
"SandboxKey": "/var/run/docker/netns/c7f74ac1d352",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "f589ece07a9ee1ab1ea20a4892ec4e266940c9c6655a420316ff0a0987d2b01d",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.36",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:24",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "4e954fb4fe031d370bf2d2aaea3bd687498f22fb9b55ff316f513a0142cab16a",
"EndpointID": "f589ece07a9ee1ab1ea20a4892ec4e266940c9c6655a420316ff0a0987d2b01d",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.36",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:24",
"DriverOpts": null
}
}
}
}
]
cat /home/dokku/<app>/nginx.conf(if applicable):- Link to the exact repository being deployed (if possible/applicable):
- If a deploy is failing or behaving unexpectedly:
- Application name
- The type of application being deployed (node, php, python, ruby, etc.)
- If using buildpacks, which custom buildpacks are in use
- If using a
Dockerfile, the contents of that file - If it exists, the contents of your
Procfile.
- Output of failing Dokku commands after running
dokku trace on(BEWARE:trace onwill print environment variables for some commands, be sure you’re not exposing any sensitive information when posting issues. You may replace these values with XXXXXXX):
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 19 (7 by maintainers)
Commits related to this issue
- feat: add container id to retire-container-failed arguments Refs dokku/dokku#3225 — committed to dokku/dokku by josegonzalez 6 years ago
- feat: add container id to retire-container-failed arguments (#3227) Refs dokku/dokku#3225 — committed to dokku/dokku by josegonzalez 6 years ago
- feat: add a way to retire old containers scheduled by the docker-local plugin These will be killed on a cron schedule, allowing us to ensure old containers don't run once memory pressure clears up - ... — committed to dokku/dokku by josegonzalez 6 years ago
- feat: add a way to retire old containers scheduled by the docker-local plugin These will be killed on a cron schedule, allowing us to ensure old containers don't run once memory pressure clears up - ... — committed to dokku/dokku by josegonzalez 6 years ago
- feat: add a way to retire old containers scheduled by the docker-local plugin These will be killed on a cron schedule, allowing us to ensure old containers don't run once memory pressure clears up - ... — committed to dokku/dokku by josegonzalez 6 years ago
- feat: add a way to retire old containers scheduled by the docker-local plugin These will be killed on a cron schedule, allowing us to ensure old containers don't run once memory pressure clears up - ... — committed to dokku/dokku by josegonzalez 6 years ago
- feat: add a way to retire old containers scheduled by the docker-local plugin These will be killed on a cron schedule, allowing us to ensure old containers don't run once memory pressure clears up - ... — committed to dokku/dokku by josegonzalez 6 years ago
Closing as there is a pull request open. Thanks for all the reports/information!
For folks who are seeing this issue, I’d like the following:
Create a gist with all this info. Thanks.