azure-sdk-for-net: [BUG] Azure.RequestFailedException: Multiple condition headers are not supported on stopping EventProcessorClient

Library name and version

Azure.Messaging.EventHubs.Processor 5.7.2

Describe the bug

Problem description

There is a Generic .NET host application that uses an EventProcessorClient to receive events from an Azure EventHub instance. Business requirements demand this EventProcessorClient to stop and start again occasionally, e.g. once in 2-3 days. Sometimes (there is no any dependency on time or anything really) when the EventProcessorClient.StopProcessingAsync(token) is called it throws an exception (details are listed in Actual behavior section)

When this exception happens only the application restart helps to start receiving events again.

Azure blobs is used for load balancing.

Expected behavior

It should stop without this exception and be able to start again when a corresponding method is called.

Actual behavior

This exception is thrown

Azure.RequestFailedException: Multiple condition headers are not supported.\nRequestId:d98ecb5a-701e-0063-0495-2b2e22000000\nTime:2023-01-18T23:33:30.6768184Z\nStatus: 400 (Multiple condition headers are not supported.)\nErrorCode: MultipleConditionHeadersNotSupported\n\nContent:\n<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>MultipleConditionHeadersNotSupported</Code><Message>Multiple condition headers are not supported.\nRequestId:d98ecb5a-701e-0063-0495-2b2e22000000\nTime:2023-01-18T23:33:30.6768184Z</Message></Error>\n\nHeaders:\nServer: Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0\nx-ms-request-id: d98ecb5a-701e-0063-0495-2b2e22000000\nx-ms-client-request-id: 6e1c8845-aa6f-4417-bb93-6f7b25d9c3fb\nx-ms-version: 2021-06-08\nx-ms-error-code: MultipleConditionHeadersNotSupported\nDate: Wed, 18 Jan 2023 23:33:30 GMT\nContent-Length: 250\nContent-Type: application/xml\n\n

With a stack trace

Azure.Storage.Blobs.BlockBlobRestClient.UploadAsync(Int64 contentLength, Stream body, Nullable`1 timeout, Byte[] transactionalContentMD5, String blobContentType, String blobContentEncoding, String blobContentLanguage, Byte[] blobContentMD5, String blobCacheControl, IDictionary`2 metadata, String leaseId, String blobContentDisposition, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, String encryptionScope, Nullable`1 tier, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, String ifMatch, String ifNoneMatch, String ifTags, String blobTagsString, Nullable`1 immutabilityPolicyExpiry, Nullable`1 immutabilityPolicyMode, Nullable`1 legalHold, CancellationToken cancellationToken)\n   at Azure.Storage.Blobs.Specialized.BlockBlobClient.UploadInternal(Stream content, BlobHttpHeaders blobHttpHeaders, IDictionary`2 metadata, IDictionary`2 tags, BlobRequestConditions conditions, Nullable`1 accessTier, BlobImmutabilityPolicy immutabilityPolicy, Nullable`1 legalHold, IProgress`1 progressHandler, String operationName, Boolean async, CancellationToken cancellationToken)\n   at Azure.Storage.Blobs.Specialized.BlockBlobClient.<>c__DisplayClass62_0.<<GetPartitionedUploaderBehaviors>b__0>d.MoveNext()\n--- End of stack trace from previous location ---\n   at Azure.Storage.PartitionedUploader`2.UploadInternal(Stream content, Nullable`1 expectedContentLength, TServiceSpecificData args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)\n   at Azure.Storage.Blobs.BlobClient.StagedUploadInternal(Stream content, BlobUploadOptions options, Boolean async, CancellationToken cancellationToken)\n   at Azure.Messaging.EventHubs.Primitives.BlobCheckpointStoreInternal.ClaimOwnershipAsync(IEnumerable`1 desiredOwnership, CancellationToken cancellationToken)\n   at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.DelegatingCheckpointStore.ClaimOwnershipAsync(IEnumerable`1 desiredOwnership, CancellationToken cancellationToken)\n   at Azure.Messaging.EventHubs.Primitives.PartitionLoadBalancer.RelinquishOwnershipAsync(CancellationToken cancellationToken)\n   at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StopProcessingInternalAsync(Boolean async, CancellationToken cancellationToken)\n   at Azure.Messaging.EventHubs.Primitives.EventProcessor`1.StopProcessingAsync(CancellationToken cancellationToken)\n

Reproduction Steps

Unfortunately there is no traceable steps that can lead to reproduction. It just happens sometimes.

Environment

• Azure Container App
• Linux OS
• .NET Core 6 (6.0.302)
• It happens in all environments we have, that includes:
  • Azure EventHub, 2 partitions, 2 Container Apps for processing, 0.5 CPU, 1 GiB each
  • Azure EventHub, 5 partitions, 5 Container Apps for processing, 1 CPU, 2 GiB each