notebook: Unable to establish websocket connection behind nginx reverse proxy

I have a Digital Ocean server running Ubuntu 14.04, with Jupyter Notebook (https://hub.docker.com/r/jupyter/notebook/) running inside a Docker container. I also have the nginx-proxy (https://github.com/jwilder/nginx-proxy) Docker container to serve as my reverse proxy.

I have it configured such that notes.rooday.com would be pointed to the Jupyter container, and that works as intended. I’m able to view the application, authenticate, make and edit files, etc. However, attempting to run an ipython notebook file gives a Connection Failed error:

A connection to the notebook server could not be established. The notebook will continue trying to reconnect, but until it does, you will NOT be able to run code. Check your network connection or notebook server configuration.

Looking inside the Chrome console, I see this error:

WebSocket connection to 'ws://notes.rooday.com/api/kernels/251adc99-3c1b-4d97-a9ae-9ecbfba3bc70/channels?session_id=28DC42BB11FD413A86D75BAE570CA379' failed: Error during WebSocket handshake: Unexpected response code: 400

The kernel attempts to reconnect, but to no avail. Checking the docker logs for the nginx proxy yields the following relevant lines:

nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:16 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:17 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"
nginx.1    | notes.rooday.com 172.17.0.1 - - [06/Apr/2016:21:42:19 +0000] "GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED HTTP/1.1" 400 34 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36"

and the logs for the Jupyter container show

[I 21:20:08.864 NotebookApp] 302 GET / (172.17.0.2) 3.53ms
[I 21:20:08.893 NotebookApp] 302 GET /tree (172.17.0.2) 4.38ms
[W 21:20:15.372 NotebookApp] 401 POST /login?next=%2Ftree (172.17.0.2) 4.28ms referer=http://notes.rooday.com/login?next=%2Ftree
[I 21:20:17.962 NotebookApp] 302 POST /login?next=%2Ftree (172.17.0.2) 4.61ms
[I 21:24:22.371 NotebookApp] 302 GET / (172.17.0.2) 2.79ms
[I 21:37:44.561 NotebookApp] 302 GET /notebooks/Untitled.ipynb (172.17.0.2) 1.37ms
[I 21:37:46.107 NotebookApp] 302 GET /tree (172.17.0.2) 1.54ms
[I 21:42:06.768 NotebookApp] 302 GET / (172.17.0.2) 4.08ms
[W 21:42:15.099 NotebookApp] 404 GET /nbextensions/widgets/extension.js?v=20160406210551 (172.17.0.2) 2.85ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[I 21:42:15.199 NotebookApp] Kernel started: 007c6099-febe-40b3-847c-5d90bcc4cf4b
[W 21:42:15.271 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20160406210551 (172.17.0.2) 9.50ms referer=http://notes.rooday.com/notebooks/Untitled.ipynb
[W 21:42:16.172 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 878.60ms referer=None
[W 21:42:17.221 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.58ms referer=None
[W 21:42:19.288 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 20.13ms referer=None
[W 21:42:23.340 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 7.98ms referer=None
[W 21:42:31.394 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 6.49ms referer=None
[W 21:42:47.459 NotebookApp] 400 GET /api/kernels/007c6099-febe-40b3-847c-5d90bcc4cf4b/channels?session_id=22F4BC7FDFCD49A28948903EA2B6ACED (172.17.0.2) 9.92ms referer=None

Based on this, and asking in the nginx-proxy issues page, it would seem that Jupyter is the cause of the problem. The nginx proxy is configured to allow websocket connections, but they’re still not going through. Has anyone else attempted to run Jupyter behind a reverse proxy, or know enough to help me find a workaround?

About this issue

Original URL
State: closed
Created 8 years ago
Reactions: 1
Comments: 26 (15 by maintainers)

Most upvoted comments

@willingc I tried following the conf but I still am not able to make a web socket connection. nginx-proxy allows mounting of a folder of conf files for specific domains that get included into the main conf, so I was able to add this to the server block for jupyter:

client_max_body_size 50M;
error_log /var/log/nginx/error.log;

location / {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 86400;
}

location ~* /(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://notes.rooday.com;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
}

The only parts from the suggested conf file that I did not include were these:

        ssl on;
        ssl_certificate /data/cert.crt;
        ssl_certificate_key /data/key.key;

        ssl_ciphers "AES128+EECDH:AES128+EDH";
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
        add_header X-Content-Type-Options nosniff;
        ssl_stapling on; # Requires nginx >= 1.3.7
        ssl_stapling_verify on; # Requires nginx => 1.3.7
        resolver_timeout 5s;

Is SSL required to get websockets working?

Edit:

Here is my jupyter_notebook_config.py file

c.NotebookApp.ip = '172.17.0.4'
c.NotebookApp.password = u'whoathatsnotsupposedtobehere'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True
c.NotebookApp.allow_origin = '*'

ROODAY on Apr 7, 2016

If you are not on CloudFlare and still have the issue. Here’s the fix: https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin ""; is the key.

jakocoo on May 25, 2017

@jakocoo

If you are not on CloudFlare and still have the issue. Here’s the fix: https://stackoverflow.com/questions/22665809/how-to-configure-ipython-behind-nginx-in-a-subpath/23912400#23912400

proxy_set_header Origin “”; is the key.

I tried that but I am still having issues with Safari saying “Kernel starting, please wait…” (which I learned happens when it can’t establish a secure websocket).

my config looks like this:

nginx jupyter.conf

location /jupyter/ {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Origin "";
}

location ~* /jupyter/(api/kernels/[^/]+/(channels|iopub|shell|stdin)|terminals/websocket)/? {
        proxy_pass http://localhost:8888;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Origin "";
}

jupyter_notebook_config.py

c.NotebookApp.allow_origin = '*'
c.NotebookApp.base_url = '/jupyter'
c.NotebookApp.open_browser = False
c.NotebookApp.password = 'sha1:REDACTED'
c.NotebookApp.port = 8888
c.NotebookApp.trust_xheaders = True

disrupted on Jul 8, 2018

@ROODAY Here’s one approach that is documented in the jupyter/docker-stacks repo: https://github.com/jupyter/docker-stacks/wiki/Docker-Recipes#running-behind-a-nginx-proxy

cc/ @parente

willingc on Apr 6, 2016

+1 I had similar issue and the problem was solved by disabling (grey-cloud) http-proxy in cloudflare.

lucaswxp on Oct 11, 2016

@willingc Nevermind, I found the issue. As it turns out, it was not related to either nginx-proxy or Jupyter, and rather Cloudflare. At the moment, Websocket connections through Cloudflare is in beta, and by default is disabled. Cloudflare recommends as a workaround to simply grey-cloud A records that make use of websocket connections, so in my case that was notes.rooday.com

https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets-

For anyone in the future attempting to use Cloudflare, Jupyter/notebook, and nginx-proxy, refer to this: https://github.com/jwilder/nginx-proxy/issues/412

Update:

Cloudflare now allows websockets for everyone, read more here.

ROODAY on May 5, 2016