dokku: Dokku restart causes container to sometimes lose old ip address
Description of problem:
My application occasionally crashes and when Dokku restarts the app, the docker IP address is not correctly mapped
Docker IP: 172.17.0.2 Nginx container IP: 172.17.0.4
This does not happen every time, fairly hard to reproduce. Externally users see 502 Gateway Problem. Nginx has some (113: No route to host) errors in the log
Output of the following commands
dokku report APP_NAME
-----> uname: Linux dokku-staging 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 16046 1173 2378 171 12495 14238
Swap: 0 0 0
-----> docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Experimental: false
-----> docker daemon info:
Containers: 11
Running: 2
Paused: 0
Stopped: 9
Images: 100
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 104
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-116-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.67GiB
Name: dokku-staging
ID: L4KE:QURH:LS6M:M734:ZVVP:MS3F:36I4:4G5L:GUAK:N2TT:Q3EY:ZKE7
Docker Root Dir: /var/lib/docker
Debug Mode (client): true
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-----> sigil version: 0.4.0
-----> herokuish version:
herokuish: 0.3.36
buildpacks:
heroku-buildpack-multi v1.0.0
heroku-buildpack-ruby v174
heroku-buildpack-nodejs v121
heroku-buildpack-clojure v78
heroku-buildpack-python v124
heroku-buildpack-java v59
heroku-buildpack-gradle v25
heroku-buildpack-scala v79
heroku-buildpack-play v26
heroku-buildpack-php v132
heroku-buildpack-go v85
heroku-buildpack-erlang fa17af9
buildpack-nginx v9
-----> dokku version: 0.11.6
-----> dokku plugins:
plugn: 0.3.0
00_dokku-standard 0.11.6 enabled dokku core standard plugin
20_events 0.11.6 enabled dokku core events logging plugin
apps 0.11.6 enabled dokku core apps plugin
build-env 0.11.6 enabled dokku core build-env plugin
certs 0.11.6 enabled dokku core certificate management plugin
checks 0.11.6 enabled dokku core checks plugin
common 0.11.6 enabled dokku core common plugin
config 0.11.6 enabled dokku core config plugin
docker-options 0.11.6 enabled dokku core docker-options plugin
domains 0.11.6 enabled dokku core domains plugin
enter 0.11.6 enabled dokku core enter plugin
git 0.11.6 enabled dokku core git plugin
letsencrypt 0.8.6 enabled Automated installation of let's encrypt TLS certificates
logs 0.11.6 enabled dokku core logs plugin
named-containers 0.11.6 enabled dokku core named containers plugin
network 0.11.6 enabled dokku core network plugin
nginx-vhosts 0.11.6 enabled dokku core nginx-vhosts plugin
plugin 0.11.6 enabled dokku core plugin plugin
proxy 0.11.6 enabled dokku core proxy plugin
ps 0.11.6 enabled dokku core ps plugin
repo 0.11.6 enabled dokku core repo plugin
shell 0.11.6 enabled dokku core shell plugin
ssh-keys 0.11.6 enabled dokku core ssh-keys plugin
storage 0.11.6 enabled dokku core storage plugin
tags 0.11.6 enabled dokku core tags plugin
tar 0.11.6 enabled dokku core tar plugin
docker inspect CONTAINER_ID(if applicable):
root@dokku-staging:~# docker inspect a3c8ec8b6da5
[
{
"Id": "a3c8ec8b6da5e93d0883de3f732f85691bb71f0ad74ed56ce1ba9905d976be7d",
"Created": "2018-04-07T01:04:05.744712134Z",
"Path": "npm",
"Args": [
"run",
"serve",
"npm",
"run",
"serve"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 10203,
"ExitCode": 0,
"Error": "",
"StartedAt": "2018-04-07T13:17:05.840788491Z",
"FinishedAt": "2018-04-07T13:17:05.414704065Z"
},
"Image": "sha256:594649afa2f1a7ff7134fee48f1f19c6edc880300df85f790d3916fcc426a38b",
"ResolvConfPath": "/var/lib/docker/containers/a3c8ec8b6da5e93d0883de3f732f85691bb71f0ad74ed56ce1ba9905d976be7d/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/a3c8ec8b6da5e93d0883de3f732f85691bb71f0ad74ed56ce1ba9905d976be7d/hostname",
"HostsPath": "/var/lib/docker/containers/a3c8ec8b6da5e93d0883de3f732f85691bb71f0ad74ed56ce1ba9905d976be7d/hosts",
"LogPath": "/var/lib/docker/containers/a3c8ec8b6da5e93d0883de3f732f85691bb71f0ad74ed56ce1ba9905d976be7d/a3c8ec8b6da5e93d0883de3f732f85691bb71f0ad74ed56ce1ba9905d976be7d-json.log",
"Name": "/sparelabs-api.web.1",
"RestartCount": 1,
"Driver": "aufs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "docker-default",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "default",
"PortBindings": {},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
},
"GraphDriver": {
"Data": null,
"Name": "aufs"
},
"Mounts": [],
"Config": {
"Hostname": "a3c8ec8b6da5",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"80/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PORT=80",
"DOKKU_APP_RESTORE=1",
"DOKKU_APP_TYPE=dockerfile",
"DOKKU_DOCKERFILE_ENTRYPOINT=ENTRYPOINT [\"npm\", \"run\", \"serve\"]",
"DOKKU_DOCKERFILE_PORTS=80",
"DOKKU_LETSENCRYPT_EMAIL=XXXXXXXXX",
"DOKKU_NGINX_SSL_PORT=443",
"DOKKU_PROXY_PORT_MAP=http:80:80 https:443:80",
"NODE_ENV=staging",
"POSTGRES_DBNAME=XXXXXXXXX",
"POSTGRES_PASSWORD=XXXXXXXX",
"POSTGRES_USER=XXXXXXX",
"UV_THREADPOOL_SIZE=8",
"DYNO=web.1",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"npm",
"run",
"serve"
],
"Image": "dokku/sparelabs-api:latest",
"Volumes": null,
"WorkingDir": "/app",
"Entrypoint": [
"npm",
"run",
"serve"
],
"OnBuild": null,
"Labels": {
"dokku": ""
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "12e1600688e8450a61d9eb8ae21cf313ee6273ea7b344b8e37498f4494f3388b",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"80/tcp": null
},
"SandboxKey": "/var/run/docker/netns/12e1600688e8",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "7ee9e3634929c404cae5efc03669e7a1d90850202c4174a677a1fc55bd2a9780",
"Gateway": "172.17.0.1",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"MacAddress": "02:42:ac:11:00:02",
"Networks": {
"bridge": {
"IPAMConfig": null,
"Links": null,
"Aliases": null,
"NetworkID": "fc2d4dbd965c696d55376aac54af665e95d928bdc4626d871e94d128a67e4b83",
"EndpointID": "7ee9e3634929c404cae5efc03669e7a1d90850202c4174a677a1fc55bd2a9780",
"Gateway": "172.17.0.1",
"IPAddress": "172.17.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:11:00:02"
}
}
}
}
]
(BEWARE: docker inspect will print environment variables for some commands, be sure you’re not exposing any sensitive information when posting issues. You may replace these values with XXXXXXX):
cat /home/dokku/<app>/nginx.conf(if applicable):
server {
listen [::]:80;
listen 80;
server_name api.staging.sparelabs.com;
access_log /var/log/nginx/sparelabs-api-access.log;
error_log /var/log/nginx/sparelabs-api-error.log;
return 301 https://$host:443$request_uri;
}
server {
listen [::]:443 ssl ;
listen 443 ssl ;
server_name api.staging.sparelabs.com;
access_log /var/log/nginx/sparelabs-api-access.log;
error_log /var/log/nginx/sparelabs-api-error.log;
ssl_certificate /home/dokku/sparelabs-api/tls/server.crt;
ssl_certificate_key /home/dokku/sparelabs-api/tls/server.key;
ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
keepalive_timeout 70;
location / {
gzip on;
gzip_min_length 1100;
gzip_buffers 4 32k;
gzip_types text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
gzip_vary on;
gzip_comp_level 6;
proxy_pass http://sparelabs-api-80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Request-Start $msec;
}
include /home/dokku/sparelabs-api/nginx.conf.d/*.conf;
error_page 400 401 402 403 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 /400-error.html;
location /400-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}
error_page 404 /404-error.html;
location /404-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}
error_page 500 501 502 503 504 505 506 507 508 509 510 511 /500-error.html;
location /500-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}
}
upstream sparelabs-api-80 {
server 172.17.0.4:80;
}
- Nginx erros
root@dokku-staging:~# dokku nginx:error-logs sparelabs-api
2018/04/09 15:31:34 [error] 5051#5051: *266340 connect() failed (113: No route to host) while connecting to upstream, client: 63.143.42.252, server: api.staging.sparelabs.com, request: "GET / HTTP/1.1", upstream: "http://172.17.0.4:80/", host: "api.staging.sparelabs.com", referrer: "https://api.staging.sparelabs.com"
2018/04/09 15:36:34 [error] 5051#5051: *266344 connect() failed (113: No route to host) while connecting to upstream, client: 63.143.42.252, server: api.staging.sparelabs.com, request: "GET / HTTP/1.1", upstream: "http://172.17.0.4:80/", host: "api.staging.sparelabs.com", referrer: "https://api.staging.sparelabs.com"
- If using a
Dockerfile, the contents of that file
FROM ubuntu:latest
ARG NODE_ENV
ENV NODE_ENV=${NODE_ENV}
# Install node.js and yarn
RUN apt-get update
RUN apt-get install -y wget tar xz-utils build-essential python
RUN cd ~ && wget https://nodejs.org/dist/v8.9.0/node-v8.9.0-linux-x64.tar.xz
RUN cd ~ && ls
RUN cd / && tar --strip-components 1 -xf ~/node-v8.9.0-linux-x64.tar.xz
RUN npm install -g yarn@1.3.2
RUN mkdir -p /app
RUN cd /app
COPY package.json yarn.lock /app/
RUN cd /app && yarn --pure-lockfile
WORKDIR /app
COPY . /app
RUN cd /app && NODE_ENV=${NODE_ENV} npm run build
EXPOSE 80
ENTRYPOINT ["npm", "run", "serve"]
Environment details (AWS, VirtualBox, physical, etc.):
DigitalOcean installation
How was Dokku installed?:
Installed an older version manually on ubuntu and then upgraded
How reproducible:
Doesnt happen everytime our app crashes
Steps to Reproduce:
- Create app and crashes periodically, set restart policy to always
- Add an external uptime monitor
- Wait for a crash to cause 502 Gateway Errors
Actual Results: Network is not correctly mapped in the Nginx file.
Expected Results: Nginx to point to correct docker IP address
Additional info:
- I am using https://github.com/dokku/dokku-letsencrypt. Not sure if that could be causing some issues as well
- Manually restarting the app fixes this issue
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 20 (12 by maintainers)
Commits related to this issue
- feat: recommend installation of the dokku-event-installer Closes #3152 — committed to dokku/dokku by josegonzalez 4 years ago
I’ve created docker-event-listener - to be packaged with dokku 0.19.0 - that should fix this issue. It still needs some testing on a live server, but it should handle properly rebuilding nginx configs for app containers that switch their IP addresses, and also will preemptively rebuild apps that have hit the maximum restart count.
@josegonzalez I’m still getting a lot of down time (“We’re sorry, but something went wrong”) with the latest dokku version 0.20.4. Restarting nginx manually fixes the issue.
What’s most weird is my containers don’t crash (and I’m using pm2 to handle processes & exceptions), so I don’t really see what it could be (one of my apps do get a lot of traffic and simultaneous websocket connections).
Is there any way to see the logs for
docker-event-listener? It seems to correctly rebuild the config, but I still have to manually runsudo service nginx restartto fix it.This is a known issue. Basically at some point, Docker decided to not re-use the same ip/port combination for containers. Why? I don’t know. Its now a “you might get the same port, you might not” sort of deal.
I’m aware of the issue and am working on a fix that involves our first persistent daemon in the core (blech) which will handle listening for these changes and restarting your nginx config.
The current fix is to re-run
dokku nginx:build-config APPfor your application when you see this.