azure-sdk-for-net: [BUG] Microsoft.Azure.ServiceBus - System.InvalidOperationException: Can't create session when the connection is closing

This issue is related to #9416 however I was asked to open a fresh thread.

Describe the bug

Intermittently for quite some time our Azure function instances running in AKS have been receiving the below exceptions coming through into Sentry.

We have a pod running .NET Core 2.2.8 with Functions v2 in our Production Kubernetes cluster and a different pod running .NET Core 3.1.5 with Functions v3 in our Sandbox cluster after recently upgrading and the exceptions are still being received from both pods intermittently. It seems to happen at random times, often days apart. I hoped upgrading to Function V3 might help to resolve the issue but alas it persists.

The production Functions pod references Microsoft.Azure.ServiceBus v3.4.0 and the sandbox Functions pod references Microsoft.Azure.ServiceBus v4.1.3.

The exception also seems to occur regardless of whether the function definition is for a Queue or Topic trigger.

Actual behavior (include Exception or Stack Trace)

Exception message: Message processing error (Action=Receive, ClientId=MessageReceiver12account-events/Subscriptions/new-account-setup, EntityPath=account-events/Subscriptions/new-account-setup, Endpoint=sndbx-sb-project-au.servicebus.windows.net)

Note: It happens with lots of different service bus queues/topics, the exception message often relates to a different queue/topic each time.

Stack Trace: System.InvalidOperationException: Can’t create session when the connection is closing. Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver”, in OnReceiveAsync Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver+<>c__DisplayClass64_0+<<ReceiveAsync>b__0>d”, in MoveNext Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “Microsoft.Azure.ServiceBus.RetryPolicy”, in RunOperation Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “Microsoft.Azure.ServiceBus.RetryPolicy”, in RunOperation Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver”, in ReceiveAsync Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver”, in ReceiveAsync Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.MessageReceivePump+<<MessagePumpTaskAsync>b__11_0>d”, in MoveNext

Another interesting piece of info, is that I am also receiving this exception as well at essentially the same time:

System.ObjectDisposedException: Cannot access a disposed object. Object name: ‘$cbs’. Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver”, in OnReceiveAsync Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver+<>c__DisplayClass64_0+<<ReceiveAsync>b__0>d”, in MoveNext Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “Microsoft.Azure.ServiceBus.RetryPolicy”, in RunOperation Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “Microsoft.Azure.ServiceBus.RetryPolicy”, in RunOperation Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver”, in ReceiveAsync Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.Core.MessageReceiver”, in ReceiveAsync Module “System.Runtime.ExceptionServices.ExceptionDispatchInfo”, in Throw Module “System.Runtime.CompilerServices.TaskAwaiter”, in ThrowForNonSuccess Module “System.Runtime.CompilerServices.TaskAwaiter”, in HandleNonSuccessAndDebuggerNotification Module “Microsoft.Azure.ServiceBus.MessageReceivePump+<<MessagePumpTaskAsync>b__11_0>d”, in MoveNext

To Reproduce Not too sure since it happens intermittently once the Function project is deployed. I have never encountered this exception when debugging locally.

An example of one of the Topic trigger function definitions is:

[FunctionName(nameof(AccountSetupTrigger))]
public async Task Run([ServiceBusTrigger("account-events", "new-account-setup", Connection = "ServiceBus")] AccountSetupMessage message, ILogger logger)

This is the csproj file (for the Sandbox Functions V3):

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
  </PropertyGroup>
  
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.0.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="4.1.2" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.7" />
    <PackageReference Include="Sentry.AspNetCore" Version="2.1.4" />
    <PackageReference Include="Stripe.net" Version="34.16.0" />
    <PackageReference Include="Xero.Api.SDK.Core" Version="1.1.4" />
  </ItemGroup>
  
  <ItemGroup>
    <ProjectReference Include="..\Common\Common.csproj" />
    <ProjectReference Include="..\Data\Data.csproj" />
  </ItemGroup>
  
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="appsettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

And the host.json file:

{
  "version": "2.0",
  "logging": {
    "logLevel": {
      "default": "Information"
    }
  },
  "extensions": {
    "serviceBus": {
      "messageHandlerOptions": {
        "maxConcurrentCalls": 1
      }
    }
  },
  "functions": [
    "AccountSetupTrigger",
    // Lots of other triggers here too...
  ]
}

Environment:

• Microsoft.Azure.ServiceBus 4.13.0 and Microsoft.Azure.ServiceBus 3.4.0
• Azure Functions V2 and V3
• .NET Core 2.2.8 and .NET Core 3.1.5
• AKS v1.14.8

Let me know if you require any more information and thanks in advance for your assistance.