prometheus: does not start up after corrupted meta.json file

Just opened a PR that would close this issue and will go in the next release. Feel free to reopen if you still experience the same issue after that.

I faced this problem also today. Finding and deleting the folder with the empty json solved the issue.

root@ip-xxx-xxx-xxx-xxx:/var/lib/kubelet/pods/ce0fba7f-c649-11e8-128370c697fc/volumes/kubernetes.io~aws-ebs/pvc-a7cf0ff9-11e8-acf2-020ae50968fe/prometheus-db# find ./ -name meta.json -size 0
./01CRRG0QYZS8GJQGS96QZYR5TJ/meta.json

@krasi-georgiev I’ll be happy to test (though, definitely not in production 😃), but crash-related bugs are notoriously hard to reproduce. I tried to check the bug using ALICE (http://research.cs.wisc.edu/adsl/Software/alice/doc/adsl-doc.html) which greatly helped me in the past and that is what I got:

Here is the write part of the test (tsdb.WriteMetaFile just calls tsdb.writeMetaFile):

package main

import (
        "math/rand"
        "os"
        "time"

        "github.com/oklog/ulid"
        "github.com/prometheus/tsdb"
)

func main() {
        now := time.Now()

        err := tsdb.WriteMetaFile(".", &tsdb.BlockMeta{
                ULID:    ulid.MustNew(ulid.Now(), rand.New(rand.NewSource(123))),
                MinTime: now.Add(-2*time.Hour).Unix() * 1000,
                MaxTime: now.Unix() * 1000,
                Version: 1,
        })
        if err != nil {
                os.Exit(1)
        }
        os.Exit(0)
}

And here the checker:

package main

import (
        "os"

        "github.com/prometheus/tsdb"
)

func main() {
        _, err := tsdb.ReadMetaFile(os.Args[1])
        if err != nil && !os.IsNotExist(err) {
                os.Exit(1)
        }
        os.Exit(0)
}

This is what I got with unmodified tsdb:

❯ alice-check --traces_dir=../traces_dir --checker=../../read-test/read-test --debug_level 1
-------------------------------------------------------------------------------
ALICE tool version 0.0.1. Please go through the documentation, particularly the
listed caveats and limitations, before deriving any inferences from this tool.
-------------------------------------------------------------------------------
Parsing traces to determine logical operations ...
Logical operations:
0       creat("meta.json.tmp", parent=107377493, mode='0666', inode=107377690)
1       append("meta.json.tmp", offset=0, count=159, inode=107377690)
2       rename(dest="meta.json", source="meta.json.tmp", source_hardlinks=1, source_parent=107377493, dest_size=0, dest_hardlinks=0, source_size=159, dest_pare
nt=107377493, source_inode=107377690, dest_inode=False)
3       fsync(".", size=60, inode=107377493)
-------------------------------------
Finding vulnerabilities...
(Dynamic vulnerability) Ordering: Operation 1 needs to be persisted before 2
(Static vulnerability) Ordering: Operation /usr/lib/golang/src/syscall/asm_linux_amd64.s:27[syscall.Syscall] needed before /usr/lib/golang/src/syscall/asm_linu
x_amd64.s:53[syscall.Syscall6]
Done finding vulnerabilities.

And this is what I got after adding f.Sync() call before f.Close in writeMetaFile:

❯ alice-check --traces_dir=../traces_dir --checker=../../read-test/read-test --debug_level 1
-------------------------------------------------------------------------------
ALICE tool version 0.0.1. Please go through the documentation, particularly the
listed caveats and limitations, before deriving any inferences from this tool.
-------------------------------------------------------------------------------
Parsing traces to determine logical operations ...
Logical operations:
0       creat("meta.json.tmp", parent=107379907, mode='0666', inode=107378562)
1       append("meta.json.tmp", offset=0, count=159, inode=107378562)
2       fsync("meta.json.tmp", size=159, inode=107378562)
3       rename(dest="meta.json", source="meta.json.tmp", source_hardlinks=1, source_parent=107379907, dest_size=0, dest_hardlinks=0, source_size=159, dest_pare
nt=107379907, source_inode=107378562, dest_inode=False)
4       fsync(".", size=60, inode=107379907)
-------------------------------------
Finding vulnerabilities...
Done finding vulnerabilities.

While this is definitely not a proof that the bug is indeed fixed, the tool has great track record and usually finds real problems.

I am not 100% sure, but logically I would say at least 5 times your biggest block. Maybe a bit less will do, but storage is not expensive these days so better be on the safe side.

btw there are plans to add storage based retention so should help use cases where storage is limited. https://github.com/prometheus/tsdb/pull/343

@Vlaaaaaaad , @bmihaescu are you sure you have enough free space?. (suggested by Brian on IRC so worth checking.) During compaction it needs a bit for temporary holder during the joining.