azure-sdk-for-net: [BUG] RenewMessageLockAsync succeeds but CompleteMessageAsync fails on when using EnableCrossEntityTransaction and processing a messages takes +10 minutes

Library name and version

Azure.Messaging.ServiceBus 7.7.0

Describe the bug

Invoking RenewMessageLockAsync works, it updates the lease but eventually when processing is completed the call to CompleteMessageAsync fails.

Azure.Messaging.ServiceBus.ServiceBusException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue, or was received by a different receiver instance. (MessageLockLost)

This while LockedUntil on message object is minutes into the future.

What is important to note is that the ServiceBusReceiver instances are the same for invoking ReceiveMessageAsync and CompleteMessageAsync but not for RenewMessageLockAsync.

Interestingly on processing duration between 5 and 10-12 minutes things just work. Sometimes it also magically works for longer processing duration.

At the moment it seems when all methods share the same ServiceBusReceiver instance no MessageLockLost occurs when invoking CompleteMessageAsync.

Is it supported to invoke RenewMessageLockAsync on a different ServiceBusReceiver instance that invoked ReceiveMessageAsync ?

Expected behavior

I would expect that when RenewMessageLockAsync is invoked and it returns a message object with a valid LockedUntil that invoking CompleteMessageAsync will not result in a MessageLockLost, even if done so on a different ServiceBusReceiver instance.

Actual behavior

Invoking CompleteMessageAsync results in the following exception if the processing duration takes roughly 12 minutes or longer.

Azure.Messaging.ServiceBus.ServiceBusException: The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue, or was received by a different receiver instance. (MessageLockLost)

Reproduction Steps

Working on this, currently got a super hacky reproduction.

Environment

Tested with .NET 6 and .NET Framework 4.8

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 35 (18 by maintainers)

Most upvoted comments

@JoshLove-msft , Sorry for the late response.

I’m presenting Thomson Reuters. Right now we are using an older version of Azure Service Bus (ASB) client -“NServiceBus.Azure.Transports.WindowsAzureServiceBus”. We are migrating to .Net 5 and want to get rid of using .Net Framework. To be able to go further we need to start using newest version of ASB client. But it contains an issue with a message renewing that blocks us from the upgrade.

Could it be backported as a patch release? If yes, could you clarify approximate dates of that patch?

This was included in 7.8.0 - https://www.nuget.org/packages/Azure.Messaging.ServiceBus/7.8.0 which was just released today.

JoshLove-msft on May 10, 2022

Actually, I think you are right. The issue only occurs when setting EnableCrossEntityTransaction to true. I suspect this is related to the fact that we have a receiver and sender link on the same session. The sender link being closed is somehow causing the receive link to also be closed, presumably because they are sharing the session.

I think this is a bug in the SDK when using EnableCrossEntityTransactions. When closing the link, we also close the session, which we should not do in this case. Instead the session cleanup should happen only when the client is disposed. Thanks @ramonsmits for helping track this down!

JoshLove-msft on Apr 22, 2022

Yes, that should work as the time limit applies only to the operations in the transaction.

JoshLove-msft on Apr 20, 2022