thanos: S3/GCS: Upload/Delete inconsistency (missing chunk file)

Thanos, Prometheus and Golang version used

• thanos: improbable/thanos:v0.4.0
  • go: 1.12.4
• prometheus: quay.io/prometheus/prometheus:v2.7.2
  • go: 1.11.5

What happened

We use an on-premise S3 store (in thanos via type: S3 in the object store config) that experienced availability issues for a 3 - 15 hour period (3 hours of common connection issues, 15 hours of less common connection issues). Multiple thanos components (shipper/sidecar, compactor) experienced timeouts while awaiting responses from the S3 service.

This resulted in the shipper/sidecar not writing complete blocks. We observed the following types of partially written blocks:

• block directory present, meta.json present, index and chunks directory missing.
• block directory present, meta.json present, index present, chunks directory missing.
• block directory present, meta.json present, index present, chunks directory present, but not all chunk files being present.

Those partially written (one might call them corrupted) blocks caused subsequent issues:

• store components failed queries over a timespan that would contain a partially written block
• compactor component crashed with non-zero exitcode (log messages before the crash below)

What you expected to happen

• sidecars do not give up in completing partially written blocks
• compactors don’t crash on partially written blocks (either skipping them or removing them outright)
• stores don’t fail queries on partially written blocks (e.g. by ignoring those blocks)

How to reproduce it (as minimally and precisely as possible):

We have been running into issues trying to build a minimal reproducible scenario. It would seem that it should be enough to have blocks with the mentioned criteria:

• block directory present, meta.json present, index and chunks directory missing.
• block directory present, meta.json present, index present, chunks directory present, but not all chunk files being present.

When trying this however we ran into the situation that these blocks did not run into the compaction plan (see here).

It seems however that once this is considered in the compaction plan and GatherIndexIssueStats is executed the compactor will fail if the index file is not present. If chunks are missing it will fail later in the prometheus tsdb compaction code.

Partial logs to relevant components

level=error
ts=2019-07-03T13:02:34.85314565Z caller=main.go:182
msg="running command failed"
err="error executing compaction: compaction failed: compaction failed for group 0@{prometheus=\"XXX-prometheus-name-XXX\",prometheus_replica=\"prometheus-thanos-system-1\"}: compact blocks [/var/thanos/store/compact/0@{prometheus=\"XXX-prometheus-name-XXX\",prometheus_replica=\"prometheus-thanos-system-1\"}/01DE9Q2EWM7Y5JZPB7V4MC4BBC /var/thanos/store/compact/0@{prometheus=\"XXX-prometheus-name-XXX\",prometheus_replica=\"prometheus-thanos-system-1\"}/01DE9XY64NVKB7Y62B9DH5A0AP /var/thanos/store/compact/0@{prometheus=\"XXX-prometheus-name-XXX\",prometheus_replica=\"prometheus-thanos-system-1\"}/01DEA4SXCNKCZ06C83Q45N6C2C /var/thanos/store/compact/0@{prometheus=\"XXX-prometheus-name-XXX\",prometheus_replica=\"prometheus-thanos-system-1\"}/01DEABNMMQFGXKXZMTKJF9T42Z]: write compaction: chunk 8 not found: reference sequence 0 out of range"

Currently this is the only log we have. Should we run into the issue again, i’ll make sure to attach more logs of the other components and cases.