prometheus: [Prometheus] Configure --web.external-url --web.route-prefix= and having a problem: Error loading available metrics!

Proposal

Use case. Why is this important? Because is something very used and there is a need of having secure under nginx.

Bug Report

Tested all possible options from internet, the only that not gives a 404 is the below one till now.

What did you expect to see? I’d expect to have the web interface GUI for prometheus and to be able to see the metrics, right now i have an message: Error loading available metrics!

What did you see instead? Under which circumstances? I’ve tried all the possible choices: https://www.robustperception.io/using-external-urls-and-proxies-with-prometheus https://www.robustperception.io/external-urls-and-path-prefixes https://prometheus.io/docs/guides/tls-encryption/#nginx-example

Environment

• System information: Linux 3.10.0-862.14.4.el7.x86_64 x86_64

• Prometheus version: 2.5.0

• Alertmanager version: 0.15.3

• Prometheus configuration file:

# cat /usr/lib/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/opt/prometheus/prometheus --config.file /opt/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus --web.console.templates=/opt/prometheus/consoles --web.external-url=http://localhost:9090/prometheus --web.route-prefix=/prometheus --web.console.libraries=/opt/prometheus/console_libraries --storage.tsdb.min-block-duration=1h --storage.tsdb.max-block-duration=12h --storage.tsdb.retention=32d --log.level=debug

[Install]
WantedBy=default.target

• Alertmanager configuration file:

# cat /usr/lib/systemd/system/alertmanager.service
[Unit]
Description=Prometheus Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/opt/alertmanager/alertmanager --config.file=/opt/alertmanager/alertmanager.yml --log.level=debug --web.external-url=http://localhost:9093/alertmanager --web.route-prefix=/alertmanager 

[Install]
WantedBy=default.target

• Logs:

From nginx server giving a curl:
[root@nginx1 ~]# curl 192.168.1.2:9090/prometheus
<a href="/prometheus/">Moved Permanently</a>.

[root@nginx1 ~]# curl 192.168.1.2:9090/prometheus/prometheus/
<a href="/prometheus/graph">Found</a>.


*Nginx config:
location /prometheus/ {
                auth_basic                    "[Monitoring] Prometheus";
                auth_basic_user_file /etc/nginx/.htpasswd;
                auth_request /_auth;
                proxy_set_header Host $host;
                proxy_pass http://prometheus_backend;
            }

upstream prometheus_backend {
server 192.168.1.2:9090;
}