thanos: Thanos receiver is unable to ingest data stating msg="Error on ingesting samples that are too old or are too far into the future"
Thanos, Prometheus and Golang version used: Thanos version: 0.19
Object Storage Provider: Azure container storage
What happened:
Receiver errors out stating Error on ingesting samples that are too old or are too far into the future. Initially I thougt error might be from prometheus with timestamps. To debug this issue, I stopped promtheues, removed all data from mount volumes as well as from remote storage. Then threw queries with quay.io/observatorium/up there also I observed same error. Now I am clueless where to see issue and need some guidance for this if this is an issue from receiver.
Exact Behaviour:
Once queries are being thrown to receiver,
- out of 3 replica, 2 replicas/3replica shows error might be
out of order sampleor “sample that too far or too old” - Memory consumption and cpu consumption goes to far like 900% and more than this
- since hpa is being used, statefulsets spins new replica
- Pushes lot of blocks to remote and filling storage pretty fast.
statefulset:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: thanos-receive
labels:
app: thanos-receive
controller.receive.thanos.io: thanos-receive-controller
controller.receive.thanos.io/hashring: default
tenant: default-tenant
namespace: monitor
spec:
podManagementPolicy: OrderedReady
replicas: 3
selector:
matchLabels:
app: thanos-receive
controller.receive.thanos.io: thanos-receive-controller
controller.receive.thanos.io/hashring: default
tenant: default-tenant
serviceName: thanos-receive-svc
template:
metadata:
labels:
app: thanos-receive
controller.receive.thanos.io: thanos-receive-controller
controller.receive.thanos.io/hashring: default
tenant: default-tenant
annotations:
sidecar.istio.io/inject: "false"
prometheus.io/scrape: 'true'
spec:
containers:
- name: thanos-receive
image: thanosio/thanos:v0.19.0
imagePullPolicy: IfNotPresent
args:
- receive
- --log.level=info
- --log.format=logfmt
- --grpc-address=0.0.0.0:10901
- --http-address=0.0.0.0:10902
- --remote-write.address=0.0.0.0:10903
- --grpc-grace-period=2m
- --tsdb.path=./thanos/receive/objstorage
- --objstore.config=$(OBJSTORE_CONFIG)
- --receive.local-endpoint=$(INSTANCE).thanos-receive-svc.monitor.svc.cluster.local:10901
- --receive.hashrings-file=/var/thanos/receive/hashring/thanos-receive-hashring.json
- --tsdb.retention=15d
- --receive.replication-factor=2
- --label=replica="$(INSTANCE)"
env:
- name: INSTANCE
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OBJSTORE_CONFIG
valueFrom:
secretKeyRef:
key: thanos-objstorage.yaml
name: thanos-objstorage-configs
livenessProbe:
httpGet:
path: /-/healthy
port: 10902
scheme: HTTP
periodSeconds: 60
#initialDelaySeconds: 30
ports:
- name: grpc-receive
containerPort: 10901
- name: http-receive
containerPort: 10902
- name: http-write
containerPort: 10903
volumeMounts:
- name: thanos-receive-controller-configmap-generated
mountPath: /var/thanos/receive/hashring
- name: receive-vc
mountPath: ./thanos/receive/objstorage
readOnly: false
resources:
requests:
cpu: 500m
memory: 1500Mi
limits:
cpu: 12
memory: 16Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
fsGroupChangePolicy: "OnRootMismatch"
serviceAccount: monitor
serviceAccountName: monitor
terminationGracePeriodSeconds: 60
volumes:
- name: thanos-receive-controller-configmap-generated
configMap:
name: thanos-receive-controller-configmap-generated
defaultMode: 420
volumeClaimTemplates:
- metadata:
name: receive-vc
labels:
app: thanos-receive
controller.receive.thanos.io: thanos-receive-controller
controller.receive.thanos.io/hashring: default
tenant: default-tenant
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: managed-premium-thanos
volumeMode: Filesystem
What you expected to happen: Receive should ingest blocks How to reproduce it (as minimally and precisely as possible):
Full logs to relevant components:
kubectl logs --tail=10 thanos-receive-9
level=warn ts=2021-05-05T18:29:26.492822082Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
level=warn ts=2021-05-05T18:29:26.492820982Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=6
level=warn ts=2021-05-05T18:29:26.492843882Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=7
level=warn ts=2021-05-05T18:29:26.492858382Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
level=warn ts=2021-05-05T18:29:26.492868982Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=4
level=warn ts=2021-05-05T18:29:26.492901482Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=7
level=warn ts=2021-05-05T18:29:26.492928182Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=5
level=warn ts=2021-05-05T18:29:26.492935882Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=5
level=warn ts=2021-05-05T18:29:26.492943482Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
level=warn ts=2021-05-05T18:29:26.493208083Z caller=writer.go:100 component=receive component=receive-writer msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=3
Anything else we need to know:
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 4
- Comments: 15 (2 by maintainers)
Please reopen this as I do not see a solution.
@kakkoyun
How can i change Prometheus setting?
If prometheus sends metric that is too old to thanos-receive, should it respond with http status 500 or 4xx? Wouldn’t 500 cause prometheus to retry endlessly?