rancher: Internal Server Error In Cluster Explorer After Upgrade to v2.5.2

I have the same issue, but I resolved it after I replace the wrong upgrade line in nginx(reverse proxy) vhost server configuration

# inside the location range
proxy_set_header Connection "Upgrade";

to

# outside the server range
map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
}

# inside the server's location range
proxy_set_header Connection $connection_upgrade;

Reference: https://www.nginx.com/blog/websocket-nginx/

I got the Cluster Explorer working. Looking at the response from that API call I realized that rancher is proxying the call to the cluster and the cluster doesn’t support websockets can’t handle websockets on this endpoint, because nginx was blindly sending websockets headers with every request:

invalid upgrade response: status code 200

The fix was to have an additional location block that did not try to upgrade the connection to websockets for /k8s EDIT: It was a little more complicated than that; with that fix I could not execute shells on containers. I had to configure nginx to set the Connection header only if it receives a websocket header from the browser.

map $http_upgrade $conn {
  default "";
  websocket "Upgrade";
}
server {
        listen 8443;
        server_name rancher.example.com;
        access_log /var/log/nginx/access.log extended;
        ssl on;
        ssl_certificate /etc/nginx/ssl/cert.pem;
        ssl_certificate_key /etc/nginx/ssl/privkey.pem;
        ssl_protocols SSLv3 TLSv1.2;
        location / {
                resolver 192.168.1.1 valid=30s;
                proxy_pass https://rancher;
                proxy_http_version 1.1;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                # websockets
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $conn;
                proxy_connect_timeout 60;
                proxy_read_timeout 240;
                error_page 502 404 /500.html;
        }
        location /500.html {
                 root /docroot/genericdown;
        }
}
server {
        listen 8080;
        server_name rancher.example.com;
        rewrite     ^   https://$server_name$request_uri? permanent;
}

I will leave the issue open for feedback as to whether or not this configuration would cause further problems (maybe I should be more specific, like set the location as /k8s/clusters?).

Also the permanent fix that would prevent a configuration like this from being necessary is for Rancher to accept websockets but proxy requests to clusters as normal HTTP handle being given websockets headers even when the endpoint doesn’t support websockets. Or to clarify that reverse proxies in front of Rancher should not use websockets at all.

Hi, I have the same problem: HTTP Error 500: from / k8s / clusters / c-nzzm4 / v1 / schemas

I started Rancher version 2.5.2

To expose rancher’s web interface I use a reverse proxy (nginx). I tried to contact the same resource (Cluster Explorer) directly from the rancher node on port 443 and the service works.