portainer: Can't get Portainer working behind nginx proxy - persistent connection closed
Description
I can’t get Portainer working behind nginx proxy. Help needed, please.
Also I’ve read two opened issues: https://github.com/portainer/portainer/issues/485 and https://github.com/portainer/portainer/issues/348 - but these threads didn’t help me.
Steps to reproduce the issue
I should note - without proxy same container works perfectly. I see all my services and all looks like right.
My own configuration
All my docker swarm services are hidden behing nginx proxy, it does additional basic authorization, white-listing, etc. So I have configuration, which is tested and works with anything (my services with websockets, Jenkins Web UI, PHPMyAdmin, Kibana, etc). So first thing that I did - just copied my configuraion:
map $http_upgrade $connection_upgrade {
'' 'close';
default 'upgrade';
}
server {
listen 80;
server_name portainer1.example.com;
gzip off;
location / {
set $upstream "portainer:80";
proxy_pass http://$upstream;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-Id $request_id;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
}
With this configuration I see login page, can login and all assets looks right. But I’ve got error, which is discussed here: persistent connection closed
.
Configuration from FAQ
So, imagine that I have too curved hands. So I take configuration from your FAQ:
server {
listen 80;
server_name portainer2.example.com;
gzip off;
location /portainer/ {
set $upstream "portainer:80";
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_pass http://$upstream/;
}
location /portainer/api/websocket/ {
set $upstream "portainer:80";
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_pass http://$upstream/api/websocket/;
}
}
Now when I open /portainer
URL - I see strange page behavior. Looks like any assets URL show index page, and app not working:
Configuration from FAQ without appending anything to URL
I’ve also tried remove additional URI prefix:
server {
listen 80;
server_name portainer3.example.com;
gzip off;
location / {
set $upstream "portainer:80";
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_pass http://$upstream/;
}
location /api/websocket/ {
set $upstream "portainer:80";
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_pass http://$upstream/api/websocket/;
}
}
Still same effect:
Technical details
- Portainer version:
1.12.2
- Portainer Docker image tag (latest/arm/windows…):
latest
- Target Docker version (the host/cluster you manage):
Client:
Version: 17.03.0-ce
API version: 1.26
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 10:53:29 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.0-ce
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 10:53:29 2017
OS/Arch: linux/amd64
Experimental: false
- Target Swarm version (if applicable): native docker swarm mode
- Platform (windows/linux): Linux, Docker under Debian Jessie
- Browser:
Mozilla Firefox 52.0.2
andOpera 44.0.2510.857
- Nginx:
1.11.12
built over officialnginx:latest
image
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 41 (7 by maintainers)
@bs-matil and anyone else who cares about running portainer at https://www.example.com/portainer/ , this works for me
I think you must add “proxy_http_version 1.1;” because the default is 1.0.
This is the config I use:
And it is working. (in fact i have 2 reverse proxy between portainer and the browser)
Hi guys, does anyone have an idea how to solve running portainer on a new url like $host:$port/portainer
With this setup i’ll only get the ui displayed but all resource calls end up in 404. Also I had to add the url rewrite as before nothing worked. Any ideas? any suggestions?
So, here is my investigation:
proxy_http_version 1.1;
- this could be defined anywhere - for example on http levelproxy_set_header Host $host:$server_port;
- not neededproxy_set_header Connection "upgrade";
withproxy_set_header Connection $connection_upgrade;
- it doesn’t workgzip
could be enabled or disabledSo, minimal working configuration is:
But I think problem is hiding here. I’ve added some logging and see this:
My configuration is based on this rule: http://nginx.org/en/docs/http/websocket.html - and when
http_upgrade
is-
- it translates toconnection_upgrade
-close
.And here is log from our own web-application, compare it:
Same rule works like a charm - connection is successfully upgraded and socket established.
So, maybe I’m wrong, but why your JS code doesn’t establish websocket connection with
Upgrade: WebSocket
header, as described here for example: https://en.wikipedia.org/wiki/HTTP/1.1_Upgrade_header ?Hi,
After passing the day on it, this is a conf working on swarm. It can surely be optimised 😃
Hi Guys
I am using Portainer on a sub-domain on local ( OSX ) and production ( Ubuntu ) because this way I can add other options ( IP, Basic access authentication ) in future. I know it works with sub-folders as well, but I always find it a little bit more difficult. In case somebody is curious, here is my config.
1. Docker on port 9000
2. /etc/nginx/nginx.conf
3. /etc/nginx/sites-enabled/portainer.yourdomain.com.conf
After
nginx -s reload
you should see Portainer.Note: For the upstream I am using
server 192.168.99.100:9000;
on OSX andserver 127.0.0.1:9000;
on Ubuntu.I have same issue, nginx can not proxy static file of portainer, so it hang, the solution below works for me, inspired by https://serverfault.com/a/989288
Using
nginx-proxy
I have solved this same issue creating a file:/etc/nginx/vhost.d/extension.domain.com_location containing:
I got it fixed by using a rewrite /docker
to
/`. This is my working configuration :Tho I am using it from another path so I don’t know if it’s helping you out. My setup looks like this:
xx.com/docker
Somewhat unrelated, but if anyone is having trouble getting proxying Portainer through Caddy to work, here’s what worked for me:
It should be
proxy_set_header Connection "upgrade";
as we discussed above.got it working with basic auth with this config:
not sure how much protection you got with this setup tho 😃
Hey guys,
I am running a portainer instance through a reverse proxy which checks for a client cert. I got everything working but the assets & exec console-
Nginx
Portainer view:
Does anyone know how to fix this? Sorry for bumbing this issue 😃
FYI bearer tokens on 1.21.0 version of Portainer looks to be so big that at least we needed to include
large_client_header_buffers 4 8k;
to NGINX config other why did see errors like these:But it also looks to be that it only happened when we had RBAC extension and LDAP authentication enabled same time.
Thanks @evertramos !
I have an environment working with NGINX Proxy and Portainer, over SSL, with auto renew certificates with Let’s Encrypt, everything with Docker.
Here are the steps to get it to work so you can compare your nginx files to see what you are missing or even use this set up.
Start this NGINX Proxy - https://github.com/evertramos/docker-compose-letsencrypt-nginx-proxy-companion
Start Poratiner from here - https://github.com/evertramos/docker-portainer-letsencrypt
Give some time to get all certificates and you can check the nginx config files.
Hope it will help!
Just in case it helps anyone else: I was having trouble getting Portainer to run behind nginx and the reason was basic auth, after I removed it Portainer ran perfectly with the config given in the faq.
@soar thanks for the investigation and the detailed report, really appreciate it.
About your last question, Portainer uses websockets for a few things only in the webapp:
I might need to review the connection/usage of the websocket in the app, but that should not affect any requests other than
/api/dockers/<endpointID>/containers/logs
or/api/dockers/<endpointID>/containers/exec
.