thanos: thanos-query: deduplication picks up time-series with missing data

Thanos, Prometheus and Golang version used thanos: v0.3.1 prometheus: v2.5.0 kubernetes: v1.12.6 Kubernetes Distro: KOPS weave: weaveworks/weave-kube:2.5.0 Cloud Platform: AWS EC2 Instance Type: R5.4XL

Architecture

      G1                               G2
      |                                |
      |                                |
      TQ1                              TQ2
      |                                |
 --------------                        |
 |------------|-------------------------                 
 |            |                        |
TSC1        TSC2                       TS
 |            |
P1           P2

G1: Grafana realtime G2: Grafana Historical TQ1: Thanos Query realtime (15d retention) TQ2: Thanos Query historical TSC: Thanos Sidecars TS: Thanos store

Each sidecar and the store is fronted by a service with *.svc.cluster.local DNS to which the --store flag points to.

G2, TQ2 are not involved in this RCA.

What happened Event Timelime:

• Due to some weave-net issues on our monitoring instance group, one of the prometheus-replicas P1 stops scraping some targets

• We See the Following metric gap in Grafana (G1) This particular metric was being scraped from cloudwatch-exporter

• We investigate thanos-query and see the following deduplication behavior:

• We can see that instead of having two series per metric we have only one, however thanos-query seems to produce contiguous data on dedup=true which is enabled by default.

• Later on we migrate the data of the bad prometheus pod on a new volume and make P2 live

• We see the following data in thanos-query with dedup=false

We can clearly see that one prometheus has data and another is missing it.

• However, when we query with dedup=true, the merged set displays missing data instead of contiguous data as expected.

What you expected to happen

We expected thanos deduplication to trust the series that has contiguous data over the one with the missing data and produce a series with contiguous data. Missing a scrape in HA Prometheus environment is expected at times, if one of the prometheus replicas has data the final output should not show missing data.

How to reproduce it (as minimally and precisely as possible):

• Have a setup as in examples of the repo in kubernetes or as described in the above architecture
• block network on one of the prometheus replicas so that it misses scrape and hence has gaps in data.
• make the blocked prometheus available again after a significant deltaT.
• use thanos-query to deduplicate the dataset and compare results.

Environment: Underlying K8S Worker Node:

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.4 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.4 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a): Linux ip-10-100-6-218 4.4.0-1054-aws #63-Ubuntu SMP Wed Mar 28 19:42:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux