prometheus: Snapshot creation error: found chunk outside of compacted

Bug Report

What did you do? Updated Prometheus v2.5.0 -> v2.6.0. What did you expect to see? A working snapshotting feature. What did you see instead? Under which circumstances? Snapshots stopped working, recovered backups from v2.5.0 aren’t working as well.

{"status":"error","errorType":"internal","error":"create snapshot: snapshot head block: write compaction: found chunk with minTime: 1547726468168 maxTime: 1547726528168 outside of compacted minTime: 1547719200000 maxTime: 1547726527867"}

Environment

• System information:

  Linux 4.15.0-43-generic x86_64

• Prometheus version:

  prometheus, version 2.6.0 (branch: HEAD, revision: dbd1d58c894775c0788470944b818cc724f550fb) build user: root@bf5760470f13 build date: 20181217-15:14:46 go version: go1.11.3

• Logs:

level=info ts=2019-01-17T12:00:00.045764719Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D0ZBFH0DZABWG7VYNNVTX9V7
level=info ts=2019-01-17T12:00:00.046032635Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D154WDRC1RJ4Z8BJPYYHCP4C
level=info ts=2019-01-17T12:00:00.046270873Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1AYA05AFGWN8AP8QCPBXP7Q
level=info ts=2019-01-17T12:00:00.046494941Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1CW28GMGNRRQDGN3DEWF0H9
level=info ts=2019-01-17T12:00:00.046644839Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGMT7Q571MBSV66MG2M55C
level=info ts=2019-01-17T12:00:00.046755629Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGKQWN3Q4K870EQTGCJ0CD
level=info ts=2019-01-17T12:00:00.046845693Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DQFF5Q5EXDTT5GRKB470SD
level=info ts=2019-01-17T12:00:27.62712416Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D0ZBFH0DZABWG7VYNNVTX9V7
level=info ts=2019-01-17T12:00:27.627376645Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D154WDRC1RJ4Z8BJPYYHCP4C
level=info ts=2019-01-17T12:00:27.627594562Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1AYA05AFGWN8AP8QCPBXP7Q
level=info ts=2019-01-17T12:00:27.627812374Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1CW28GMGNRRQDGN3DEWF0H9
level=info ts=2019-01-17T12:00:27.627949074Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGMT7Q571MBSV66MG2M55C
level=info ts=2019-01-17T12:00:27.628038373Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGKQWN3Q4K870EQTGCJ0CD
level=info ts=2019-01-17T12:00:27.628112874Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DQFF5Q5EXDTT5GRKB470SD