portainer: Can't get Portainer working behind nginx proxy - persistent connection closed

Description

I can’t get Portainer working behind nginx proxy. Help needed, please.

Also I’ve read two opened issues: https://github.com/portainer/portainer/issues/485 and https://github.com/portainer/portainer/issues/348 - but these threads didn’t help me.

Steps to reproduce the issue

I should note - without proxy same container works perfectly. I see all my services and all looks like right.

My own configuration

All my docker swarm services are hidden behing nginx proxy, it does additional basic authorization, white-listing, etc. So I have configuration, which is tested and works with anything (my services with websockets, Jenkins Web UI, PHPMyAdmin, Kibana, etc). So first thing that I did - just copied my configuraion:

map $http_upgrade $connection_upgrade {
    ''                      'close';
    default                 'upgrade';    
}

server {
    listen 80;
    server_name portainer1.example.com;
    gzip off;

    location / {
        set $upstream "portainer:80";
        proxy_pass http://$upstream;

        proxy_redirect off;
        proxy_buffering off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Request-Id $request_id;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}

With this configuration I see login page, can login and all assets looks right. But I’ve got error, which is discussed here: persistent connection closed.

default

Configuration from FAQ

So, imagine that I have too curved hands. So I take configuration from your FAQ:

server {
    listen 80;
    server_name portainer2.example.com;
    gzip off;

    location /portainer/ {
        set $upstream "portainer:80";
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://$upstream/;
    }

    location /portainer/api/websocket/ {
        set $upstream "portainer:80";
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        proxy_pass http://$upstream/api/websocket/;
    }
}

Now when I open /portainer URL - I see strange page behavior. Looks like any assets URL show index page, and app not working:

default

Configuration from FAQ without appending anything to URL

I’ve also tried remove additional URI prefix:

server {
    listen 80;
    server_name portainer3.example.com;
    gzip off;

    location / {
        set $upstream "portainer:80";
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://$upstream/;
    }

    location /api/websocket/ {
        set $upstream "portainer:80";
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        proxy_pass http://$upstream/api/websocket/;
    }
}

Still same effect:

default

Technical details

  • Portainer version: 1.12.2
  • Portainer Docker image tag (latest/arm/windows…): latest
  • Target Docker version (the host/cluster you manage):
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:53:29 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:53:29 2017
 OS/Arch:      linux/amd64
 Experimental: false
  • Target Swarm version (if applicable): native docker swarm mode
  • Platform (windows/linux): Linux, Docker under Debian Jessie
  • Browser: Mozilla Firefox 52.0.2 and Opera 44.0.2510.857
  • Nginx: 1.11.12 built over official nginx:latest image

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 41 (7 by maintainers)

Commits related to this issue

Most upvoted comments

@bs-matil and anyone else who cares about running portainer at https://www.example.com/portainer/ , this works for me

  • ubuntu 18.04
  • nginx 1.14.0-0ubuntu1.2
  • docker.io 18.06.1-0ubuntu1.2~18.04.1
  • portainer 1.20.2
	location /portainer {
		rewrite ^/portainer(/.*)$ /$1 break;
		proxy_pass http://localhost:9000/;
		proxy_http_version 1.1;
		proxy_set_header Connection "";
	}

	location /portainer/api {
		proxy_set_header Upgrade $http_upgrade;
		proxy_pass http://localhost:9000/api;
		proxy_set_header Connection 'upgrade';
		proxy_http_version 1.1;
	}

I think you must add “proxy_http_version 1.1;” because the default is 1.0.

This is the config I use:

server {
   listen 80;
   server_name portainer.example.com;
   
   client_max_body_size 8m;
   ignore_invalid_headers off;
   
   location / {
     proxy_pass http://10.1.2.3:9876;
     proxy_http_version 1.1;
     proxy_set_header   Host               $host:$server_port;
     proxy_set_header   X-Real-IP          $remote_addr;
     proxy_set_header   X-Forwarded-For    $proxy_add_x_forwarded_for;
     proxy_set_header   X-Forwarded-Proto  $scheme;
     proxy_set_header   Upgrade            $http_upgrade;
     proxy_set_header   Connection         "upgrade";
   }
   
}

And it is working. (in fact i have 2 reverse proxy between portainer and the browser)

Hi guys, does anyone have an idea how to solve running portainer on a new url like $host:$port/portainer

server {

  client_max_body_size 8m;   
   ignore_invalid_headers off;

   location /portainer/ {
       rewrite ^/portainer(.*) /$1 break
       proxy_pass http://docker-portainer;
       proxy_http_version 1.1;
       proxy_set_header Connection "";

   }

   location /portainer/api/websocket/ {
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";
         proxy_http_version 1.1;
         proxy_pass http://docker-portainer/api/websocket/;
  }
}

With this setup i’ll only get the ui displayed but all resource calls end up in 404. Also I had to add the url rewrite as before nothing worked. Any ideas? any suggestions?

So, here is my investigation:

  1. proxy_http_version 1.1; - this could be defined anywhere - for example on http level
  2. proxy_set_header Host $host:$server_port; - not needed
  3. If I replace proxy_set_header Connection "upgrade"; with proxy_set_header Connection $connection_upgrade; - it doesn’t work
  4. gzip could be enabled or disabled

So, minimal working configuration is:

server {
    listen 80;
    server_name portainer.example.com;

    location / {
        set $upstream "portainer:80";
        proxy_pass http://$upstream;
        proxy_set_header   Upgrade            $http_upgrade;
        # You should always do "upgrade"
        proxy_set_header   Connection         "upgrade";
    }
}

But I think problem is hiding here. I’ve added some logging and see this:

[03/Apr/2017:11:20:56 +0000] "GET /images/ie-spacer.gif HTTP/1.1"   304 0    "/css/app.1868c70b.css"   "Mozilla/5.0" "-" http_upgrade: "-", connection_upgrade "close"
[03/Apr/2017:11:20:56 +0000] "GET /api/docker/1/networks HTTP/1.1"  500 39   "/"                       "Mozilla/5.0" "-" http_upgrade: "-", connection_upgrade "close"

My configuration is based on this rule: http://nginx.org/en/docs/http/websocket.html - and when http_upgrade is - - it translates to connection_upgrade - close.

And here is log from our own web-application, compare it:

[03/Apr/2017:11:44:21 +0000] "GET / HTTP/1.1"                                                                                                                                  304 0   "-" "Mozilla/5.0" "-" test http_upgrade: "-", connection_upgrade: "close"
[03/Apr/2017:11:44:21 +0000] "GET /socket.io/?__sails_io_sdk_version=1.1.4&__sails_io_sdk_platform=node&__sails_io_sdk_language=javascript&EIO=3&transport=websocket HTTP/1.1" 101 574 "-" "Mozilla/5.0" "-" test http_upgrade: "websocket", connection_upgrade: "upgrade"

Same rule works like a charm - connection is successfully upgraded and socket established.

So, maybe I’m wrong, but why your JS code doesn’t establish websocket connection with Upgrade: WebSocket header, as described here for example: https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header ?

Hi,

After passing the day on it, this is a conf working on swarm. It can surely be optimised 😃

server {

	# Docker DNS resolver (needed for upstream)
	resolver 127.0.0.11 valid=5s ipv6=off;
	listen       80;
	server_name  portainer.domain.ovh;

	access_log off;
	error_log  /var/log/nginx/error.log  warn;

	location / {
		set $upstream_endpoint http://portainer:9000;
		proxy_http_version 1.1;
		proxy_set_header   Connection "";
		proxy_set_header   Host $host;
		proxy_set_header   X-Forwarded-Host $server_name;
		add_header         X-Upstream $upstream_addr;
		proxy_pass $upstream_endpoint;
	}

	location /api/websocket/ {
		set $upstream_endpoint http://portainer:9000;
		proxy_buffering off;
		proxy_set_header   Upgrade $http_upgrade;
		proxy_set_header   Connection "Upgrade";
		proxy_set_header   Host $host;
		proxy_set_header   X-Forwarded-Server $host;
		proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header   X-Forwarded-Host $server_name;
		add_header            X-Upstream $upstream_addr;
		proxy_http_version 1.1;
		proxy_pass $upstream_endpoint;
                # Need this for the console
		proxy_redirect http://portainer:9000 $scheme://$host/;
	}

}

Hi Guys

I am using Portainer on a sub-domain on local ( OSX ) and production ( Ubuntu ) because this way I can add other options ( IP, Basic access authentication ) in future. I know it works with sub-folders as well, but I always find it a little bit more difficult. In case somebody is curious, here is my config.

1. Docker on port 9000

portainer/portainer .... 0.0.0.0:9000->9000/tcp

2. /etc/nginx/nginx.conf

http {
    
    // your config here

    include /etc/nginx/sites-enabled/*.conf;
}

3. /etc/nginx/sites-enabled/portainer.yourdomain.com.conf

upstream portainer_yourdomain_com {
    server 127.0.0.1:9000;
}

server {

    listen 80;
    server_name portainer.yourdomain.com;
    
    location / {

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_cache_bypass $http_upgrade;
        proxy_cache_valid 200 302 1m;
        proxy_pass $scheme://portainer_yourdomain_com;
    }
}

After nginx -s reload you should see Portainer.

Note: For the upstream I am using server 192.168.99.100:9000; on OSX and server 127.0.0.1:9000; on Ubuntu.

I have same issue, nginx can not proxy static file of portainer, so it hang, the solution below works for me, inspired by https://serverfault.com/a/989288

location ~* ^/portainer(/.*)$ {
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            proxy_pass http://127.0.0.1:9000$1;
}

location ~* ^/portainer/(api/websocket/.*)$ {
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_http_version 1.1;
            proxy_pass http://127.0.0.1:9000$1;
}

Using nginx-proxy I have solved this same issue creating a file:

/etc/nginx/vhost.d/extension.domain.com_location containing:

proxy_http_version 1.1;
proxy_set_header Connection "upgrade";

I got it fixed by using a rewrite /docker to/`. This is my working configuration :Screenshot_20210411-135610_Termius.jpg

Tho I am using it from another path so I don’t know if it’s helping you out. My setup looks like this: xx.com/docker

Somewhat unrelated, but if anyone is having trouble getting proxying Portainer through Caddy to work, here’s what worked for me:

https://portainer.domain.tld, http://portainer.domain.tld {
    proxy / upstream:9000 {
        transparent
        header_upstream Upgrade "upgrade"
        header_upstream X-Upstream upstream:9000
    }
    tls cert key
}

containing: proxy_http_version 1.1; proxy_set_header Connection “”;

It should be proxy_set_header Connection "upgrade"; as we discussed above.

got it working with basic auth with this config:


set $portainer portainer:9000;

location /docker/api/websocket/ {
      rewrite /docker/api/websocket/(.*) /api/websocket/$1 break;
      proxy_pass http://$portainer;
      
      proxy_set_header upgrade $http_upgrade;
      proxy_set_header connection 'upgrade';
      proxy_http_version 1.1;
      
      # THIS LINE fixes basic auth
      proxy_set_header authorization "";
}

# THIS BLOCK needs to be separate too
location /docker/api/ {
      rewrite /docker/api/(.*) /api/$1 break;
      proxy_pass http://$portainer;
}

# HERE you can enable basic auth
location /docker/ {
      auth_basic           "Login required";
      auth_basic_user_file /etc/nginx/.htpasswd;
      
      rewrite /docker/(.*) /$1 break;
      proxy_pass http://$portainer;
}

not sure how much protection you got with this setup tho 😃

Hey guys,

I am running a portainer instance through a reverse proxy which checks for a client cert. I got everything working but the assets & exec console-

Nginx

server {
    listen 443 ssl http2;
    server_name xxx.xxx.team;

    ssl on;
    ssl_certificate /etc/ssl/certs/cloudflare-origin.crt;
    ssl_certificate_key /etc/ssl/private/cloudflare-origin.key;
    ssl_client_certificate /etc/ssl/certs/cloudflare-client.crt;
    ssl_verify_client on;
    proxy_redirect off;

    location / {
       proxy_http_version 1.1;
       proxy_pass http://data:9000;
       proxy_set_header   Host $host;
       proxy_set_header   Upgrade            $http_upgrade;
       proxy_set_header   Connection         "upgrade";
    }
}

Portainer view:

WmNcOO2 1

Does anyone know how to fix this? Sorry for bumbing this issue 😃

FYI bearer tokens on 1.21.0 version of Portainer looks to be so big that at least we needed to include large_client_header_buffers 4 8k; to NGINX config other why did see errors like these: image

But it also looks to be that it only happened when we had RBAC extension and LDAP authentication enabled same time.

I have an environment working with NGINX Proxy and Portainer, over SSL, with auto renew certificates with Let’s Encrypt, everything with Docker.

Here are the steps to get it to work so you can compare your nginx files to see what you are missing or even use this set up.

  1. Start this NGINX Proxy - https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion

  2. Start Poratiner from here - https://github.com/evertramos/docker-portainer-letsencrypt

Give some time to get all certificates and you can check the nginx config files.

Hope it will help!

Just in case it helps anyone else: I was having trouble getting Portainer to run behind nginx and the reason was basic auth, after I removed it Portainer ran perfectly with the config given in the faq.

@soar thanks for the investigation and the detailed report, really appreciate it.

About your last question, Portainer uses websockets for a few things only in the webapp:

  • Tailing container logs
  • Container exec (container console)

I might need to review the connection/usage of the websocket in the app, but that should not affect any requests other than /api/dockers/<endpointID>/containers/logs or /api/dockers/<endpointID>/containers/exec.