thanos: Compact deduplication and retention is not working as expected

Thanos, Prometheus and Golang version used: Thanos 0.22.0 Golang go1.16.6

Object Storage Provider: S3 (Minio)

What happened: We have two Prometheus servers with the exact same configuration to scrape the targets, both with the same cluster and different replica external label and a compactor with the below flags:

          --compact.enable-vertical-compaction
          --deduplication.replica-label="replica"
          --deduplication.func=penalty
          --retention.resolution-raw=20d
          --retention.resolution-5m=90d

What you expected to happen: See only one block of data for a time range and no raw chunk older than 20 days, but here is the status of the blocks: screenshot_4 This block is obviously older than 20 days but it is still present in the object store: screenshot_5

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 23 (11 by maintainers)

Most upvoted comments

Replica label should be without “”

evel=info ts=2021-09-22T05:13:01.945232765Z caller=compact.go:241 msg="deduplication.replica-label specified, enabling vertical compaction" dedupReplicaLabels=replica
level=info ts=2021-09-22T05:13:01.945344035Z caller=compact.go:246 msg="vertical compaction is enabled" compact.enable-vertical-compaction=true

....
level=debug ts=2021-09-22T05:13:03.689823716Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689871928Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689897056Z caller=fetcher.go:717 msg="replica label removed" label=replica
level=debug ts=2021-09-22T05:13:03.689905195Z caller=fetcher.go:717 msg="replica label removed" label=replica
....

jnauska on Sep 22, 2021

@yeya24 here is the list flags the compactor is running with:


          - compact

          - --wait 

          - --wait-interval=5m

          - --consistency-delay=30m

          - --compact.enable-vertical-compaction

          - --deduplication.replica-label="replica"

          - --deduplication.func=penalty

          - --retention.resolution-raw=20d

          - --retention.resolution-5m=28d

          - --delete-delay=30m

          - --compact.concurrency=2

          - --data-dir=/var/thanos/compact

          - |

            --objstore.config=type: S3

Can you enable the debug log?

yeya24 on Sep 4, 2021