nginx-proxy-manager: n-p-m container doesn't come back up after docker service restart after updating to 2.10.0

Checklist

  • Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
    • Yes
  • Are you sure you’re not using someone else’s docker image?
    • Yes
  • Have you searched for similar issues (both open and closed)?
    • Yes

Describe the bug

For years I have been backing up my rpi4 on which n-p-m runs with https://github.com/framps/raspiBackup and everything was fine. Before the backup I shut down docker service, then after backup restart it and all the containers came properly back up. After updating to 2.10.0 the container doesn’t start back up when docker service starts with the error in log below.

Nginx Proxy Manager Version

2.10.0

To Reproduce Steps to reproduce the behavior:

  1. sudo systemctl stop docker
  2. run my backup
  3. sudo systemctl start docker && sudo systemctl daemon-reload (daemon-reload as the backup stops bunch of other services like cron and snmpd and they always complain about the daemon-reload when they come back up)
  4. See error

Expected behavior

container should come back up as normal (all the others do)

Screenshots

n/a

Operating System

raspberry pi4 Linux pi-docker 6.1.19-v8+ #1637 SMP PREEMPT Tue Mar 14 11:11:47 GMT 2023 aarch64 GNU/Linux

Additional context

logs:

nginx-proxy-manager_pi-docker  | [3/28/2023] [8:25:01 AM] [SSL      ] › ℹ  info      Renew Complete
nginx-proxy-manager_pi-docker  | s6-rc: info: service legacy-services: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service legacy-services successfully stopped
nginx-proxy-manager_pi-docker  | s6-rc: info: service nginx: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service frontend: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service backend: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service frontend successfully stopped
nginx-proxy-manager_pi-docker  | s6-rc: info: service backend successfully stopped
nginx-proxy-manager_pi-docker  | s6-rc: info: service nginx successfully stopped
nginx-proxy-manager_pi-docker  | s6-rc: info: service prepare: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service prepare successfully stopped
nginx-proxy-manager_pi-docker  | s6-rc: info: service legacy-cont-init: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service legacy-cont-init successfully stopped
nginx-proxy-manager_pi-docker  | s6-rc: info: service fix-attrs: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service fix-attrs successfully stopped
nginx-proxy-manager_pi-docker  | s6-rc: info: service s6rc-oneshot-runner: stopping
nginx-proxy-manager_pi-docker  | s6-rc: info: service s6rc-oneshot-runner successfully stopped
nginx-proxy-manager_pi-docker  | [3/28/2023] [8:30:52 AM] [Global   ] › ℹ  info      PID 140 received SIGTERM
nginx-proxy-manager_pi-docker  | [3/28/2023] [8:30:52 AM] [Global   ] › ℹ  info      Stopping.
nginx-proxy-manager_pi-docker  | s6-rc: info: service s6rc-oneshot-runner: starting
nginx-proxy-manager_pi-docker  | s6-rc: info: service s6rc-oneshot-runner successfully started
nginx-proxy-manager_pi-docker  | s6-rc: info: service fix-attrs: starting
nginx-proxy-manager_pi-docker  | s6-rc: info: service fix-attrs successfully started
nginx-proxy-manager_pi-docker  | s6-rc: info: service legacy-cont-init: starting
nginx-proxy-manager_pi-docker  | s6-rc: info: service legacy-cont-init successfully started
nginx-proxy-manager_pi-docker  | s6-rc: info: service prepare: starting
nginx-proxy-manager_pi-docker  | s6-rc: fatal: timed out
nginx-proxy-manager_pi-docker  | s6-sudoc: fatal: unable to get exit status from server: Operation timed out
nginx-proxy-manager_pi-docker  | /run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
[root@docker-7bce5741f82f:/app]# cat /run/uncaught-logs/current
cat: /run/uncaught-logs/current: No such file or directory

docker-compose.yml:

version: '3'
services:
  nginx-proxy-manager:
    container_name: nginx-proxy-manager_pi-docker
    image: 'jc21/nginx-proxy-manager:latest'
    ports:
      - '10080:80'
      - '10081:81'
      - '10443:443'
    volumes:
      - ./data_npm:/data
      - ./letsencrypt:/etc/letsencrypt
    restart: always
    environment:
      X_FRAME_OPTIONS: "sameorigin"
      DB_SQLITE_FILE: "/data/database.sqlite"
      TZ: "Europe/London"
      PUID: "1000"
      PGID: "1000"

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Reactions: 8
  • Comments: 39

Most upvoted comments

Same here, 16 pages down for minimum 10h. Did you test it beofre release?

I had the same error, removing my PGID and PUID solved it for me

Hello again. I answer myself alone and by the way I leave the note in case someone else has the problem. Although it seemed untrue, the solution is to restart the docker several times until it finishes booting. It looks like a car in winter that doesn’t want to start at first.

This is the complete log of reboots and what it does at each boot.

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service prepare: starting
❯ Configuring npmuser ...
id: 'npmuser': no such user
s6-rc: fatal: timed out
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

# Restart Docker 1

s6-sudoc: fatal: unable to get exit status from server: Operation timed out
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service prepare: starting
❯ Configuring npmuser ...
undefined 
usermod: no changes
s6-rc: fatal: timed out
s6-sudoc: fatal: unable to get exit status from server: Operation timed out
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.

# Restart Docker 2

s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service prepare: starting
❯ Configuring npmuser ...
undefined 
usermod: no changes
❯ Checking paths ...
❯ Setting ownership ...
s6-rc: fatal: timed out
/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information.
s6-sudoc: fatal: unable to get exit status from server: Operation timed out

# Restart Docker 3

s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service prepare: starting
❯ Configuring npmuser ...
undefined 
usermod: no changes
❯ Checking paths ...
❯ Setting ownership ...
❯ Dynamic resolvers ...
❯ IPv6 ...
Disabling IPV6 in hosts in: /etc/nginx/conf.d
- /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
- /etc/nginx/conf.d/include/force-ssl.conf
- /etc/nginx/conf.d/include/proxy.conf
- /etc/nginx/conf.d/include/block-exploits.conf
- /etc/nginx/conf.d/include/ssl-ciphers.conf
- /etc/nginx/conf.d/include/assets.conf
- /etc/nginx/conf.d/include/ip_ranges.conf
- /etc/nginx/conf.d/include/resolvers.conf
- /etc/nginx/conf.d/production.conf
- /etc/nginx/conf.d/default.conf
Disabling IPV6 in hosts in: /data/nginx
- /data/nginx/proxy_host/1.conf
- /data/nginx/proxy_host/10.conf
- /data/nginx/proxy_host/17.conf
- /data/nginx/proxy_host/16.conf
- /data/nginx/proxy_host/4.conf
- /data/nginx/proxy_host/8.conf
- /data/nginx/proxy_host/7.conf
- /data/nginx/proxy_host/9.conf
- /data/nginx/proxy_host/3.conf
- /data/nginx/proxy_host/14.conf
- /data/nginx/proxy_host/2.conf
- /data/nginx/proxy_host/5.conf
- /data/nginx/redirection_host/1.conf
- /data/nginx/redirection_host/2.conf
- /data/nginx/custom/server_proxy.conf
- /data/nginx/default_host/site.conf
❯ Docker secrets ...

-------------------------------------
 _   _ ____  __  __
| \ | |  _ \|  \/  |
|  \| | |_) | |\/| |
| |\  |  __/| |  | |
|_| \_|_|   |_|  |_|
-------------------------------------
User ID:  0
Group ID: 0
-------------------------------------

s6-rc: info: service prepare successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service frontend: starting
s6-rc: info: service backend: starting
s6-rc: info: service nginx successfully started
❯ Starting nginx ...
s6-rc: info: service backend successfully started
s6-rc: info: service frontend successfully started
s6-rc: info: service legacy-services: starting
❯ Starting backend ...
s6-rc: info: service legacy-services successfully started
[4/3/2023] [8:53:22 PM] [Global   ] › ℹ  info      Using MySQL configuration
[4/3/2023] [8:53:26 PM] [Migrate  ] › ℹ  info      Current database version: 20211108145214
[4/3/2023] [8:53:27 PM] [Setup    ] › ℹ  info      Logrotate Timer initialized
[4/3/2023] [8:53:28 PM] [Setup    ] › ℹ  info      Logrotate completed.
[4/3/2023] [8:53:28 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
[4/3/2023] [8:53:28 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[4/3/2023] [8:53:28 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
[4/3/2023] [8:53:28 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
[4/3/2023] [8:53:29 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
[4/3/2023] [8:53:29 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[4/3/2023] [8:53:29 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
[4/3/2023] [8:53:29 PM] [Global   ] › ℹ  info      Backend PID 138 listening on port 3000 ...
[4/3/2023] [8:53:31 PM] [Nginx    ] › ℹ  info      Reloading Nginx
[4/3/2023] [8:53:31 PM] [SSL      ] › ℹ  info      Renew Complete

# Boot ok, restart docker 4 (verify ok reboot)

s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service nginx: stopping
s6-rc: info: service frontend: stopping
s6-rc: info: service backend: stopping
s6-rc: info: service frontend successfully stopped
s6-rc: info: service backend successfully stopped
s6-rc: info: service nginx successfully stopped
s6-rc: info: service prepare: stopping
s6-rc: info: service prepare successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped
[4/3/2023] [8:54:25 PM] [Global   ] › ℹ  info      PID 138 received SIGTERM
[4/3/2023] [8:54:25 PM] [Global   ] › ℹ  info      Stopping.
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service prepare: starting
❯ Configuring npmuser ...
undefined 
usermod: no changes
❯ Checking paths ...
❯ Setting ownership ...
❯ Dynamic resolvers ...
❯ IPv6 ...
Disabling IPV6 in hosts in: /etc/nginx/conf.d
- /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
- /etc/nginx/conf.d/include/force-ssl.conf
- /etc/nginx/conf.d/include/proxy.conf
- /etc/nginx/conf.d/include/block-exploits.conf
- /etc/nginx/conf.d/include/ssl-ciphers.conf
- /etc/nginx/conf.d/include/assets.conf
- /etc/nginx/conf.d/include/ip_ranges.conf
- /etc/nginx/conf.d/include/resolvers.conf
- /etc/nginx/conf.d/production.conf
- /etc/nginx/conf.d/default.conf
Disabling IPV6 in hosts in: /data/nginx
- /data/nginx/proxy_host/1.conf
- /data/nginx/proxy_host/10.conf
- /data/nginx/proxy_host/17.conf
- /data/nginx/proxy_host/16.conf
- /data/nginx/proxy_host/4.conf
- /data/nginx/proxy_host/8.conf
- /data/nginx/proxy_host/7.conf
- /data/nginx/proxy_host/9.conf
- /data/nginx/proxy_host/3.conf
- /data/nginx/proxy_host/14.conf
- /data/nginx/proxy_host/2.conf
- /data/nginx/proxy_host/5.conf
- /data/nginx/redirection_host/1.conf
- /data/nginx/redirection_host/2.conf
- /data/nginx/custom/server_proxy.conf
- /data/nginx/default_host/site.conf
❯ Docker secrets ...

-------------------------------------
 _   _ ____  __  __
| \ | |  _ \|  \/  |
|  \| | |_) | |\/| |
| |\  |  __/| |  | |
|_| \_|_|   |_|  |_|
-------------------------------------
User ID:  0
Group ID: 0
-------------------------------------

s6-rc: info: service prepare successfully started
s6-rc: info: service nginx: starting
s6-rc: info: service frontend: starting
s6-rc: info: service backend: starting
s6-rc: info: service nginx successfully started
s6-rc: info: service frontend successfully started
❯ Starting nginx ...
s6-rc: info: service backend successfully started
s6-rc: info: service legacy-services: starting
❯ Starting backend ...
s6-rc: info: service legacy-services successfully started
[4/3/2023] [8:55:14 PM] [Global   ] › ℹ  info      Using MySQL configuration
[4/3/2023] [8:55:18 PM] [Migrate  ] › ℹ  info      Current database version: 20211108145214
[4/3/2023] [8:55:18 PM] [Setup    ] › ℹ  info      Logrotate Timer initialized
[4/3/2023] [8:55:20 PM] [Setup    ] › ℹ  info      Logrotate completed.
[4/3/2023] [8:55:20 PM] [IP Ranges] › ℹ  info      Fetching IP Ranges from online services...
[4/3/2023] [8:55:20 PM] [IP Ranges] › ℹ  info      Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[4/3/2023] [8:55:21 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v4
[4/3/2023] [8:55:21 PM] [IP Ranges] › ℹ  info      Fetching https://www.cloudflare.com/ips-v6
[4/3/2023] [8:55:21 PM] [SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
[4/3/2023] [8:55:21 PM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[4/3/2023] [8:55:21 PM] [IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
[4/3/2023] [8:55:21 PM] [Global   ] › ℹ  info      Backend PID 139 listening on port 3000 ...
[4/3/2023] [8:55:23 PM] [Nginx    ] › ℹ  info      Reloading Nginx
[4/3/2023] [8:55:24 PM] [SSL      ] › ℹ  info      Renew Complete

# Restart Docker OK

@coalfield Don’t have time to look at it. But this (bind() to 0.0.0.0:80 failed (13: Permission denied)) issue seems to be discussed here with a reason and (temporary) solution: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/2750#issuecomment-1488381509

@zandhaas Port mapping doesn’t relate to the issue, since it’s external to the container. This permission error became an issue because access to port 80 on the inside needs a root user in the current configuration. And I’m using a seperate non-root docker user for all of my containers.

New error, is it just me?

nginx: [warn] the “user” directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:4 nginx: [emerg] bind() to 0.0.0.0:80 failed (13: Permission denied)

Weirdly the routes still (mostly) worked but the front-end was completely reset. I had no user accounts and it showed zero proxy hosts or streams even though they were in the back-end and routing correctly for the most part (Vikunja threw a hissy fit and I needed to restart the container twice in the last 12 hours or so to get it to work again).

Found this and downgrade to 2.9.22 immediately corrected the issue. I can’t pull logs right now as I’m off on vacation and happy to have at least found a stable state.

@hugalafutro @Mosney This seems to be expected behavior.

Release notes for 2.9.22 state: https://github.com/NginxProxyManager/nginx-proxy-manager/releases

v2.9.22

⚠️ The next release v2.10.0 will have changes that may adversely affect you, please pin your docker tag to 2.9.22 and upgrade after you conduct a full backup.

and

v2.10.0

⚠️ This release has changes that may adversely affect you, please pin your docker tag to the previous release 2.9.22 and upgrade after you conduct a full backup.

But I can’t find anything about how to prevent this ‘adverse affect’.