prometheus: opening storage failed: found unsequential head chunk files 23 and 25

What did you do?

Exit prometheus cleanly

level=warn ts=2020-06-17T18:14:44.525Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2020-06-17T18:14:44.525Z caller=main.go:547 msg="Stopping scrape discovery manager..."
level=info ts=2020-06-17T18:14:44.525Z caller=main.go:561 msg="Stopping notify discovery manager..."
level=info ts=2020-06-17T18:14:44.525Z caller=main.go:583 msg="Stopping scrape manager..."                          
level=info ts=2020-06-17T18:14:44.525Z caller=main.go:543 msg="Scrape discovery manager stopped"                
level=info ts=2020-06-17T18:14:44.525Z caller=main.go:557 msg="Notify discovery manager stopped"                       
level=info ts=2020-06-17T18:14:44.525Z caller=manager.go:888 component="rule manager" msg="Stopping rule manager..."
level=info ts=2020-06-17T18:14:44.525Z caller=manager.go:898 component="rule manager" msg="Rule manager stopped"       
level=info ts=2020-06-17T18:14:44.525Z caller=main.go:577 msg="Scrape manager stopped"                                                
level=info ts=2020-06-17T18:14:44.531Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."               
level=info ts=2020-06-17T18:14:44.531Z caller=main.go:749 msg="Notifier manager stopped"
level=info ts=2020-06-17T18:14:44.531Z caller=main.go:761 msg="See you next time!"

What did you expect to see?

I can restart it

What did you see instead? Under which circumstances?

level=info ts=2020-06-17T18:14:51.603Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" durat
ion=15d
level=info ts=2020-06-17T18:14:51.603Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.19.0, branch=master, revision=86dc8
9a39aefa5f8f64af25c22f2055137ba3423)"
level=info ts=2020-06-17T18:14:51.603Z caller=main.go:338 build_context="(go=go1.15beta1, user=roidelapluie@oxygen, date=20200617-18:00:10)
"
level=info ts=2020-06-17T18:14:51.603Z caller=main.go:339 host_details="(Linux 5.6.8-roidelapluie #1 SMP Thu Apr 30 23:23:52 CEST 2020 x86_
64 oxygen (none))"
level=info ts=2020-06-17T18:14:51.604Z caller=main.go:340 fd_limits="(soft=1024, hard=102400)"
level=info ts=2020-06-17T18:14:51.604Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-06-17T18:14:51.605Z caller=main.go:678 msg="Starting TSDB ..."
level=info ts=2020-06-17T18:14:51.605Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-06-17T18:14:51.605Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592352000000 maxt=1592359200000 u
lid=01EB0KB79S5NGARYRX25316A4Z
level=info ts=2020-06-17T18:14:51.605Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592373600000 maxt=1592380800000 u
lid=01EB137TZWK06KAVX4AKYV3Z1Q
level=info ts=2020-06-17T18:14:51.605Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592337196974 maxt=1592352000000 u
lid=01EB137V4XCWYST9E87QAM1NJF
level=info ts=2020-06-17T18:14:51.605Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592395200000 maxt=1592402400000 u
lid=01EB1PSKC7D1AYFMMAEHVZ39V3
level=info ts=2020-06-17T18:14:51.607Z caller=main.go:547 msg="Stopping scrape discovery manager..."
level=info ts=2020-06-17T18:14:51.607Z caller=main.go:561 msg="Stopping notify discovery manager..."
level=info ts=2020-06-17T18:14:51.607Z caller=main.go:583 msg="Stopping scrape manager..."
level=info ts=2020-06-17T18:14:51.607Z caller=main.go:557 msg="Notify discovery manager stopped"
level=info ts=2020-06-17T18:14:51.607Z caller=main.go:543 msg="Scrape discovery manager stopped"
level=info ts=2020-06-17T18:14:51.607Z caller=manager.go:888 component="rule manager" msg="Stopping rule manager..."
level=info ts=2020-06-17T18:14:51.607Z caller=manager.go:898 component="rule manager" msg="Rule manager stopped"
level=info ts=2020-06-17T18:14:51.607Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
level=info ts=2020-06-17T18:14:51.607Z caller=main.go:749 msg="Notifier manager stopped"
level=info ts=2020-06-17T18:14:51.607Z caller=main.go:577 msg="Scrape manager stopped"
level=error ts=2020-06-17T18:14:51.608Z caller=main.go:758 err="opening storage failed: found unsequential head chunk files 23 and 25"

Environment

System information:

Linux 5.6.8-roidelapluie x86_64
Prometheus version:

prometheus, master / 86dc89a39aefa5f8f64af25c22f2055137ba3423 build user: roidelapluie@oxygen build date: 20200617-18:00:10 go version: go1.15beta1

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 21 (13 by maintainers)

Most upvoted comments

Hey!

I’m still seeing this with prometheus 2.19.1.

Steps to reproduce:

download prometheus 2.19.1 and unarchive it
copy my data directory in the unarchived directory
start prometheus with the default configuration

11:53:17 ~/Downloads/prometheus-2.19.1.linux-amd64 $ ./prometheus 
level=info ts=2020-06-19T09:54:20.848Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2020-06-19T09:54:20.848Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.19.1, branch=HEAD, revision=eba3fdcbf0d378b66600281903e3aab515732b39)"
level=info ts=2020-06-19T09:54:20.848Z caller=main.go:338 build_context="(go=go1.14.4, user=root@62700b3d0ef9, date=20200618-16:35:26)"
level=info ts=2020-06-19T09:54:20.848Z caller=main.go:339 host_details="(Linux 5.6.16-1-MANJARO #1 SMP PREEMPT Wed Jun 3 14:26:28 UTC 2020 x86_64 flixbook (none))"
level=info ts=2020-06-19T09:54:20.848Z caller=main.go:340 fd_limits="(soft=1024, hard=524288)"
level=info ts=2020-06-19T09:54:20.848Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-06-19T09:54:20.849Z caller=web.go:524 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-06-19T09:54:20.849Z caller=main.go:678 msg="Starting TSDB ..."
level=info ts=2020-06-19T09:54:20.849Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1576389600000 maxt=1578139200000 ulid=01DY3MC0F2A56WPGK2S6EYSXSS
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1578139200000 maxt=1578722400000 ulid=01DY9XA8MXS0SHJBBMBKGB5XC0
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1578722400000 maxt=1579305600000 ulid=01DYZQSQ8Z9F1KR9RTYQZD4QNB
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1579305600000 maxt=1579888800000 ulid=01E11CF4ANKRS8D31YZXABJ9PQ
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1579888800000 maxt=1580472000000 ulid=01E11CF4RXP64T6Y4NKFDGC084
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1580472000000 maxt=1581055200000 ulid=01E11CF57KEP88N2THMGE7WP2E
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1581055200000 maxt=1581638400000 ulid=01E11CF5P0E1H1QYFRFSBATGYJ
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1581638400000 maxt=1582221600000 ulid=01E1J6DD6G2VSQ7MWB4ZW55VDE
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1582221600000 maxt=1582804800000 ulid=01E23JK7RBXZF4C4RQA30RMX61
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1582804800000 maxt=1583388000000 ulid=01E2MYS3KC80Q2CQFXF0C4F86X
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1583388000000 maxt=1583971200000 ulid=01E36AYYFV9TCTKRCEN5N21ZDH
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1583971200000 maxt=1584554400000 ulid=01E3QQ4QH6236J019B36RZXDCH
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1584554400000 maxt=1585137600000 ulid=01E493AMC4XP367PQ0Z2ZNC15J
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1585137600000 maxt=1585720800000 ulid=01E4TFGFMW1V4BWQJTBFGS5YQ9
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1585720800000 maxt=1586304000000 ulid=01E5BVPB8QA6QX8YY4REHS3J6N
level=info ts=2020-06-19T09:54:20.850Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1586304000000 maxt=1586887200000 ulid=01E5X7W6JGH1W6G02CYYS3C935
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1586887200000 maxt=1587470400000 ulid=01E6EM1ZF0F58NZZD3EKT54MMV
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1587470400000 maxt=1588053600000 ulid=01E7007TASB9KEARS2Z1Z5VWRM
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1588053600000 maxt=1588636800000 ulid=01E7HCDRMVF4CTPMF3MKSTVXVC
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1588636800000 maxt=1589220000000 ulid=01E82RKMGMYQRGAP55R6SN6CGQ
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1589220000000 maxt=1589803200000 ulid=01E8M4SEMKM4YS8BB3CJXKS260
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1589803200000 maxt=1590386400000 ulid=01E95GYZRBWWJMSD01YPJ6B2JW
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1590386400000 maxt=1590969600000 ulid=01E9PX4TZ1FJF4WGP32Y6NK8SD
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1590969600000 maxt=1591552800000 ulid=01EA89AQJC0VW5MEQ9GW4CEKW1
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1591552800000 maxt=1592136000000 ulid=01EASNGS84DV99NZBSDWTAS7FN
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592136000000 maxt=1592330400000 ulid=01EAZEXEEPTW1G7D3QDE3MZF02
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592330400000 maxt=1592524800000 ulid=01EB58A1X5Z3HFMQZNM7Y2E217
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592546400000 maxt=1592553600000 ulid=01EB5WX68JDA507D8ZWVK78H0Y
level=info ts=2020-06-19T09:54:20.851Z caller=repair.go:59 component=tsdb msg="Found healthy block" mint=1592524800000 maxt=1592546400000 ulid=01EB5WX70T1YWK4C4326EFRYQ0
level=info ts=2020-06-19T09:54:20.853Z caller=main.go:547 msg="Stopping scrape discovery manager..."
level=info ts=2020-06-19T09:54:20.853Z caller=main.go:561 msg="Stopping notify discovery manager..."
level=info ts=2020-06-19T09:54:20.854Z caller=main.go:583 msg="Stopping scrape manager..."
level=info ts=2020-06-19T09:54:20.854Z caller=main.go:557 msg="Notify discovery manager stopped"
level=info ts=2020-06-19T09:54:20.854Z caller=main.go:543 msg="Scrape discovery manager stopped"
level=info ts=2020-06-19T09:54:20.854Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..."
level=info ts=2020-06-19T09:54:20.854Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped"
level=info ts=2020-06-19T09:54:20.854Z caller=main.go:577 msg="Scrape manager stopped"
level=info ts=2020-06-19T09:54:20.854Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..."
level=info ts=2020-06-19T09:54:20.854Z caller=main.go:749 msg="Notifier manager stopped"
level=error ts=2020-06-19T09:54:20.854Z caller=main.go:758 err="opening storage failed: found unsequential head chunk files 5 and 7"

If it helps I would be willing to send you my data directory (422 MB) in a non-public way.

unixlab on Jun 19, 2020