rancher: If you're seeing this Grafana has failed to load its application files

Hi,

It appears to be related to the unconditional proxy_set_header Connection "upgrade"; Seems Grafana does not expect it?!

You can make the Connection header reflects the client one, in your nginx config, add in http scope:

map $http_upgrade $connection_upgrade {
	default upgrade;
	''	close;
}

Then replace the proxy_set_header Connection "upgrade" with proxy_set_header $connection_upgrade.

Hope this helps.

Regards,

Ben.

In my case, I just faced this error after upgrading rancher to 2.4.5 version. Since I’ve installed rancher on a single node docker environment with no proxy, I cannot apply @rutgerFH’s comments.

Fortunately, I found the new experimental proxy feature was enabled in Global Settings. Just after I deactivating the proxy feature, Grafana works again without error.

FYI, the experimental proxy feature prevents Rancher’s tools like monitoring, logging, Istio, etc. See, https://rancher.com/docs/rancher/v2.x/en/installation/options/feature-flags/

And you can turn it off with ease. See, https://rancher.com/docs/rancher/v2.x/en/installation/options/feature-flags/#disabling-features-with-the-rancher-ui

good luck

It is supposed to be: proxy_set_header Connection $connection_upgrade

You forgot the Connection in between

ie: `map $http_upgrade $connection_upgrade { default Upgrade; ‘’ close; }

server { # SSL configuration listen 443 ssl http2;

    server_name some.domain.com;
    
    # 443 logging
    error_log /etc/nginx/logs/change_error_443.log warn;
    access_log /etc/nginx/logs/changeaccess_443.log;

    location / {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-Port $server_port;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://some.domain.com:80;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header X-Real-IP $remote_addr;
            
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Server $host;
            
            proxy_pass_request_headers on;
            add_header X-location websocket always;
            
            proxy_read_timeout 900s;
            #proxy_ssl_verify off;
            proxy_buffering off;
    }

server { listen 80;

    server_name some.domain.com;

    #logging:
    error_log /etc/nginx/logs/change_error.log warn;
    access_log /etc/nginx/logs/changeaccess.log;
    
    # redirect to https
    location / {
            return 301 https://$host$request_uri;
    }

} `

I solved this problem. October 19, 2023

• grafana version 10.1.2 (but it doesn’t seem to have any effect on version.)

I installed grafana 10.1.2 on AWS EC2 and used it well for two weeks. In the 3rd week, tuning was done to modify the favicon and welcome message. The task was to change js files in the public folder. In the 4th week, when accessing the gafana page, an error page with proxy information etc. suddenly appeared and I could not access it.

I searched a lot on Google, but there was a lot of proxy information. I disconnected the domain record from route53 and connected with the IP address, but the error message was the same.

Therefore, I thought it was not a proxy problem.

I installed grafana on a new ec2 and copied and overwritten the configuration files.

/usr/share/grafana/public/ tar -cvfz public_origin public

And then grafana-server was restarted. And I was connected to the grafana domain as before.

If you modified the js file, it may be helpful to restore the settings like this.

Good luck.

It appears that grafana is unaware that its’ operating under a subdomain. Despite hacking a things into place to make at least the URL for a single cluster pass correctly, I was unable to make it work, and thus will give up on it. https://github.com/rancher/rancher/issues/25035 may be related.