thanos: receive: Incorrect error status code with multiple AlreadyExists
Thanos, Prometheus and Golang version used:
Thanos: quay.io/thanos/thanos:v0.26.0 Prometheus: quay.io/prometheus/prometheus:2.36.0
Object Storage Provider: Azure
What happened:
I am running a multi-tenant router-receiver setup with multiple Prometheus instances remote writing to it, and a replication factor of 2. At times an error occurs where remote writing fails, and Prometheus will have to resend time series. However when this occurs some time series may already be present in the receiver store and return a AlreadyExistserror. This does not seem to be an issue when a single receiver already has the time series, but a bigger issue when multiple receivers returns the same error. The Prometheus instance will receive a 500 error and try again, forcing it to end up in an infinite loop where it will never be able to recover from.
What you expected to happen:
I expect the correct error to be returned for this condition so that the Prometheus instance will stop sending the same time series which already exist in the receiver store.
How to reproduce it (as minimally and precisely as possible):
This error will only occur when receivers are being torn down and setup again one by one. While this is being done some replications may fail causing the Prometheus instance to resend the same data. It is hard to pin point more conditions to get this error to happen, as sometimes everything works out without any issues. When the error occurs every single Prometheus instance will fail to write as they will continue to attempt sending the same metrics over and over again.
Full logs to relevant components:
level=error ts=2022-06-07T14:16:31.548131665Z caller=handler.go:373 component=receive component=receive-handler tenant=*** err="2 errors: replicate write request for endpoint thanos-ingestor-receiver-0.thanos-ingestor-receiver.monitor.svc.cluster.local:10901: quorum not reached: forwarding request to endpoint thanos-ingestor-receiver-0.thanos-ingestor-receiver.monitor.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = store locally for endpoint thanos-ingestor-receiver-0.monitor.svc.cluster.local:10901: conflict; replicate write request for endpoint thanos-ingestor-receiver-5.thanos-ingestor-receiver.monitor.svc.cluster.local:10901: quorum not reached: forwarding request to endpoint thanos-ingestor-receiver-0.thanos-ingestor-receiver.monitor.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = store locally for endpoint thanos-ingestor-receiver-0.monitor.svc.cluster.local:10901: conflict" msg="internal server error"
Anything else we need to know:
I have done some analysis of what is occurring and my guess is that the determineWriteErrorCause function has a bug in it where it does not detect the conflict error and return it.
https://github.com/thanos-io/thanos/blob/17c576472d80972bfd3705e1e0a08e6f8da8e04b/pkg/receive/handler.go#L730
Looking at the handler code it is apparent that if a wrapped error is returned a 500 status code will be given instead of a 409 which is the correct one for conflict errors. https://github.com/thanos-io/thanos/blob/17c576472d80972bfd3705e1e0a08e6f8da8e04b/pkg/receive/handler.go#L361-L375
Reading through the replication code it seems like the original AlreadyExists error is being wrapped two or three times into other errors. When this is done the function to determine if the error is a conflict will not work and instead it will return the whole error causing a 500 status.
https://github.com/thanos-io/thanos/blob/22aefd6cc666d0b509ed3748d6f719b60952cf6b/pkg/receive/handler.go#L484
https://github.com/thanos-io/thanos/blob/22aefd6cc666d0b509ed3748d6f719b60952cf6b/pkg/receive/handler.go#L582
https://github.com/thanos-io/thanos/blob/22aefd6cc666d0b509ed3748d6f719b60952cf6b/pkg/receive/handler.go#L665
We need to write a unit test specifically replicating the AlreadyExists error to verify that determineWriteErrorCause will return a conflict error.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 2
- Comments: 15 (7 by maintainers)
Can confirm, we see the same issue with a replication factor of 1
I think I have found the origin of this issue. As the comment in
determineWriteErrorCausestates it will not go deeper than the first multi error to determine the cause. The issue is that there are places where a returned multi error is flattened and places where it is not. If the multi error is not flattened the cause will never be determined. Resulting in a 500 status code.This error is flattened with a comment explaining why it is. https://github.com/thanos-io/thanos/blob/22aefd6cc666d0b509ed3748d6f719b60952cf6b/pkg/receive/handler.go#L510-L513
In the case where replicate is called in fan out it is not flattened, meaning that there is a risk that a multi error is placed inside of a multi error. https://github.com/thanos-io/thanos/blob/22aefd6cc666d0b509ed3748d6f719b60952cf6b/pkg/receive/handler.go#L475-L490
It might take some time to setup a new environment without the ingestor split. I will try to get some time to write a unit test to reproduce this.