thanos: Thanos receiver is unable to ingest data stating msg="Error on ingesting samples that are too old or are too far into the future"

Thanos, Prometheus and Golang version used: Thanos version: 0.19

Object Storage Provider: Azure container storage

What happened: Receiver errors out stating Error on ingesting samples that are too old or are too far into the future. Initially I thougt error might be from prometheus with timestamps. To debug this issue, I stopped promtheues, removed all data from mount volumes as well as from remote storage. Then threw queries with quay.io/observatorium/up there also I observed same error. Now I am clueless where to see issue and need some guidance for this if this is an issue from receiver. Exact Behaviour: Once queries are being thrown to receiver,

  1. out of 3 replica, 2 replicas/3replica shows error might be out of order sample or “sample that too far or too old”
  2. Memory consumption and cpu consumption goes to far like 900% and more than this
  3. since hpa is being used, statefulsets spins new replica
  4. Pushes lot of blocks to remote and filling storage pretty fast.

statefulset:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-receive
  labels:
    app: thanos-receive
    controller.receive.thanos.io: thanos-receive-controller
    controller.receive.thanos.io/hashring: default
    tenant: default-tenant
  namespace: monitor
spec:
  podManagementPolicy: OrderedReady
  replicas: 3
  selector:
    matchLabels:
      app: thanos-receive
      controller.receive.thanos.io: thanos-receive-controller
      controller.receive.thanos.io/hashring: default
      tenant: default-tenant
  serviceName: thanos-receive-svc
  template:
    metadata:
      labels:
        app: thanos-receive
        controller.receive.thanos.io: thanos-receive-controller
        controller.receive.thanos.io/hashring: default
        tenant: default-tenant
      annotations: 
        sidecar.istio.io/inject: "false"
        prometheus.io/scrape: 'true'
    spec:
      containers:
        - name: thanos-receive
          image: thanosio/thanos:v0.19.0
          imagePullPolicy: IfNotPresent
          args:
            - receive 
            - --log.level=info
            - --log.format=logfmt
            - --grpc-address=0.0.0.0:10901
            - --http-address=0.0.0.0:10902
            - --remote-write.address=0.0.0.0:10903
            - --grpc-grace-period=2m         
            - --tsdb.path=./thanos/receive/objstorage
            - --objstore.config=$(OBJSTORE_CONFIG)
            - --receive.local-endpoint=$(INSTANCE).thanos-receive-svc.monitor.svc.cluster.local:10901
            - --receive.hashrings-file=/var/thanos/receive/hashring/thanos-receive-hashring.json
            - --tsdb.retention=15d
            - --receive.replication-factor=2
            - --label=replica="$(INSTANCE)"
          env:
          - name: INSTANCE
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: OBJSTORE_CONFIG
            valueFrom:
              secretKeyRef:
                key: thanos-objstorage.yaml
                name: thanos-objstorage-configs
          livenessProbe:
            httpGet:
              path: /-/healthy
              port: 10902
              scheme: HTTP
            periodSeconds: 60
              #initialDelaySeconds: 30
          ports:
            - name: grpc-receive
              containerPort: 10901
            - name: http-receive
              containerPort: 10902
            - name: http-write
              containerPort: 10903
          volumeMounts:
          - name: thanos-receive-controller-configmap-generated
            mountPath: /var/thanos/receive/hashring
          - name: receive-vc
            mountPath: ./thanos/receive/objstorage
            readOnly: false
          resources:
            requests:
              cpu: 500m
              memory: 1500Mi
            limits:
              cpu: 12
              memory: 16Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: FallbackToLogsOnError
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 2000
        runAsGroup: 2000
        runAsNonRoot: true
        runAsUser: 2000
        fsGroupChangePolicy: "OnRootMismatch"
      serviceAccount: monitor
      serviceAccountName: monitor
      terminationGracePeriodSeconds: 60
      volumes: 
      - name: thanos-receive-controller-configmap-generated
        configMap: 
          name: thanos-receive-controller-configmap-generated
          defaultMode: 420
  volumeClaimTemplates: 
  - metadata: 
      name: receive-vc
      labels: 
        app: thanos-receive
        controller.receive.thanos.io: thanos-receive-controller
        controller.receive.thanos.io/hashring: default
        tenant: default-tenant
    spec: 
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: managed-premium-thanos 
      volumeMode: Filesystem

What you expected to happen: Receive should ingest blocks How to reproduce it (as minimally and precisely as possible):

Full logs to relevant components:

kubectl logs --tail=10 thanos-receive-9
level=warn ts=2021-05-05T18:29:26.492822082Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
level=warn ts=2021-05-05T18:29:26.492820982Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=6
level=warn ts=2021-05-05T18:29:26.492843882Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=7
level=warn ts=2021-05-05T18:29:26.492858382Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
level=warn ts=2021-05-05T18:29:26.492868982Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=4
level=warn ts=2021-05-05T18:29:26.492901482Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=7
level=warn ts=2021-05-05T18:29:26.492928182Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=5
level=warn ts=2021-05-05T18:29:26.492935882Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=5
level=warn ts=2021-05-05T18:29:26.492943482Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
level=warn ts=2021-05-05T18:29:26.493208083Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3

Anything else we need to know:

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 4
  • Comments: 15 (2 by maintainers)

Most upvoted comments

Please reopen this as I do not see a solution.

@kakkoyun

  1. I verified timestamps from Prometheus and receive, both are in the same time zone and reflecting the same time.
  2. If I am considering it rightly, I saw some discussion over backfilling data and understood that receive cant take 2 hr lagging data from Prometheus. so I made changes in Prometheus setting to send 1 hr time window data.

@kakkoyun

  1. I verified timestamps from Prometheus and receive, both are in the same time zone and reflecting the same time.
  2. If I am considering it rightly, I saw some discussion over backfilling data and understood that receive cant take 2 hr lagging data from Prometheus. so I made changes in Prometheus setting to send 1 hr time window data.

How can i change Prometheus setting?

If prometheus sends metric that is too old to thanos-receive, should it respond with http status 500 or 4xx? Wouldn’t 500 cause prometheus to retry endlessly?