prometheus: Web UI - Error loading available metrics!

Bug Report

What did you do? Successfully upgraded Prometheus 2.0 to 2.15.0-rc0 and then open WEB UI.

What did you expect to see? I expect /graph page without error and with working search suggest drop-down list.

What did you see instead? Under which circumstances?

Open Web UI behind Apache proxy on https://prometheus.company.org/ via Google Chrome.
I am forwarded to https://prometheus.company.org/graph page where I see “Error loading available metrics!” error.
Search suggest drop-down list is not working, there are no suggestions.

Environment

System information:

Linux 3.10.0-1062.4.1.el7.x86_64 x86_64

Prometheus version:

prometheus, version 2.15.0-rc.0 (branch: HEAD, revision: 61f1953c50148b3857bad6ab9a5622e1f1277695)
  build user:       root@67bb57f8015b
  build date:       20191218-17:10:23
  go version:       go1.13.5

Prometheus parameters:

/opt/prometheus/prometheus --storage.tsdb.retention.time=60d --storage.tsdb.retention.size=98GB --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data --web.external-url=https://prometheus.company.org/ --web.console.templates=/opt/prometheus/consoles --query.timeout=1m --web.read-timeout=1m --web.route-prefix=/

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 19 (12 by maintainers)

Commits related to this issue

Handle V1 indexes, some of which have unsorted posting offset tables. Fixes #6535 Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> — committed to prometheus/prometheus by brian-brazil 4 years ago
Handle V1 indexes, some of which have unsorted posting offset tables. (#6564) Fixes #6535 Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> — committed to prometheus/prometheus by brian-brazil 4 years ago
Handle V1 indexes, some of which have unsorted posting offset tables. (#6564) Fixes #6535 Signed-off-by: Brian Brazil <brian.brazil@robustperception.io> — committed to prometheus/prometheus by brian-brazil 4 years ago

Most upvoted comments

Thank you. After upgrade to the latest version of Prometheus is everything good.

Rohlik on Jan 8, 2020

@brian-brazil here you have what you need 😃

meta.json:

{
        "version": 1,
        "ulid": "01DS1P0FNEAM2YPR8DAKCACNDJ",
        "minTime": 1572890400000,
        "maxTime": 1573084800000,
        "stats": {
                "numSamples": 343318557,
                "numSeries": 54892,
                "numChunks": 2861464
        },
        "compaction": {
                "level": 4,
                "sources": [
                        "01DRW3F8TFKRG0JYWVH908TA8Y",
                        "01DRWAB02PH017ZB9ZEQHX5X38",
                        "01DRWH6QATKYJ8S6HXP665NF25",
                        "01DRWR2EJFJM21WKM8GQ405APT",
                        "01DRWYY5TP3R2YNNQGK60JQMKQ",
                        "01DRX5SX3F62WHSYZ2NVYXC4MD",
                        "01DRXCNMABS6A4ZQ6AYXPX26PR",
                        "01DRXKHBKRZ0AXT2W3CK377R9W",
                        "01DRXTD2TCT1BVCD52RP4B337K",
                        "01DRY18T327QE2GAQH1P49V5DE",
                        "01DRY84HANDP7G549NWBB3WCFB",
                        "01DRYF08J50XH6EBC1B0RFSHRG",
                        "01DRYNVZTV50A6FJKNQE5S8QJE",
                        "01DRYWQQ2S842T024J37BYYT2G",
                        "01DRZ3KEB17CERPWRTASRN285S",
                        "01DRZAF5JN6CTBBXWR0KSTKDMG",
                        "01DRZHAWV3MSH0CE4SC1Y22QE6",
                        "01DRZR6M34CT87MYANMSBXDKVT",
                        "01DRZZ2BA5ZRVCZ00PR95BTJ9F",
                        "01DS05Y2J6QJHRBNN9P3GJCKAF",
                        "01DS0CSSTHBDGPRN32MWRWJT7Z",
                        "01DS0KNH2BZ5N828SX5EDM2XCA",
                        "01DS0TH8ACKW736WAD9SQ21K0B",
                        "01DS11CZJMK2Q35WHEC7EAFDH9",
                        "01DS188PTKE5VQ4C8XVZJCJ4RA",
                        "01DS1F4E32AAG8SDCYDDTNF7RZ",
                        "01DS1P05AFF8DPYM2XZY2XKVJV"
                ]
        }
}

index file have 27MB, so I uploaded it on GDrive - https://drive.google.com/file/d/1NXQbZqBDRqJwCPzUzgEI3BIeVUgEdUIR/view?usp=sharing

The time range in meta.json is related to time before upgrade to higher version.

Rohlik on Jan 4, 2020

@Rohlik Thanks!

For @brian-brazil’s questions, could you share data/01DS1P0FNEAM2YPR8DAKCACNDJ/meta.json and data/01DS1P0FNEAM2YPR8DAKCACNDJ/index (careful, might contain sensitive data if your metrics are sensitive), as well as whether the time range in the block’s meta.json file happened before or after the upgrade to 2.15?

juliusv on Jan 4, 2020