azure-sdk-for-net: [BUG] SNAT Port Exhaustion with Azure Service Bus

Describe the bug I’m using a Linux Web App that connects to an Azure Service Bus to send messages. The web app is causing SNAT port exhaustion connecting to servicebus.windows.net. It seems connections to servicebus.windows.net are not being reused or closed correctly. Messages are received in another app service. We send under a few hundred messages per hour.

Expected behavior Service Bus connections are reused or closed after use.

Actual behavior (include Exception or Stack Trace) The app is having issues with SNAT port exhaustion causing crashing.

To Reproduce In startup.cs, registration:

services.AddAzureClients(clientsBuilder =>
                {
                    clientsBuilder.AddServiceBusClient(Configuration.GetValue<string>("Queue:ServiceBusConnectionString"))
                      // (Optional) Provide name for instance to retrieve by with DI
                      //.WithName("Client1Name")
                      // (Optional) Override ServiceBusClientOptions (e.g. change retry settings)
                      .ConfigureOptions(options =>
                      {
                          //options.RetryOptions.Delay = TimeSpan.FromMilliseconds(50);
                          options.RetryOptions.MaxDelay = TimeSpan.FromSeconds(5);
                          options.RetryOptions.MaxRetries = 3;
                      });
                });

All queue message creation goes through this class:

    public class QueueService : IQueueService
    {
        private readonly IOptions<AppSettings> options;
        private readonly ILogger<QueueService> logger;
        private readonly ServiceBusClient serviceBusClient;

        public QueueService(IOptions<AppSettings> options, ILogger<QueueService> logger, ServiceBusClient serviceBusClient)
        {
            this.options = options;
            this.logger = logger;
            this.serviceBusClient = serviceBusClient;
        }

        public async Task Enqueue(string queue, string label, string message, bool throwErrors = false)
        {
            if ((options.Value?.Queue?.ServiceBusConnectionString).IsNullOrEmpty())
            {
                logger.LogError($"No service bus connection set for {label}: {message}");
                if (throwErrors)
                {
                    throw new Exception($"No service bus connection set for {label}: {message}");
                }
                return;
            }
            try
            {
                //https://docs.microsoft.com/en-us/dotnet/api/overview/azure/messaging.servicebus-readme-pre
                //Sender should not be disposed
                var sender = serviceBusClient.CreateSender(queue);
                
                var queueMessage = new ServiceBusMessage(Encoding.UTF8.GetBytes(message));
                queueMessage.Subject = label;
                await sender.SendMessageAsync(queueMessage);
            }
            catch (Exception e)
            {
                logger.LogError(e, $"Queue Error for {label}: {message}");
                if (throwErrors)
                {
                    throw;
                }
            }
        }

Environment: Assembly Azure.Messaging.ServiceBus, Version=7.4.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8 Azure Web App on Linux App Service .NET 5.0

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 29 (14 by maintainers)

Most upvoted comments

This seems to be working better. We’re deploying to test and will deploy to prod later today. Thank you again for your help!

services.AddSingleton<IQueueSenderCache, QueueSenderCache>();

    public class QueueService : IQueueService
    {
        private readonly IOptions<AppSettings> options;
        private readonly ILogger<QueueService> logger;
        private readonly IQueueSenderCache queueSenderCache;

        public QueueService(IOptions<AppSettings> options, ILogger<QueueService> logger, IQueueSenderCache queueSenderCache)
        {
            this.options = options;
            this.logger = logger;
            this.queueSenderCache = queueSenderCache;
        }

        public async Task Enqueue(string queue, string label, string message, bool throwErrors = false)
        {
            if ((options.Value?.Queue?.ServiceBusConnectionString).IsNullOrEmpty())
            {
                logger.LogError($"No service bus connection set for {label}: {message}");
                if (throwErrors)
                {
                    throw new Exception($"No service bus connection set for {label}: {message}");
                }
                return;
            }
            try
            {
                var sender = queueSenderCache.GetSender(queue);
                
                var queueMessage = new ServiceBusMessage(Encoding.UTF8.GetBytes(message));
                queueMessage.Subject = label;
                await sender.SendMessageAsync(queueMessage);
            }
            catch (Exception e)
            {
                logger.LogError(e, $"Queue Error for {label}: {message}");
                if (throwErrors)
                {
                    throw;
                }
            }
        }


    }

public class QueueSenderCache : IQueueSenderCache
    {
        private readonly ServiceBusClient serviceBusClient;
        public ConcurrentDictionary<string, ServiceBusSender> serviceBuses = new ConcurrentDictionary<string, ServiceBusSender>();

        //WARNING:SINGLETON
        public QueueSenderCache(ServiceBusClient serviceBusClient)
        {
            this.serviceBusClient = serviceBusClient;
        }

        public ServiceBusSender GetSender(string queueOrTopicName)
        {
            //https://docs.microsoft.com/en-us/dotnet/api/overview/azure/messaging.servicebus-readme-pre
            //Sender should not be disposed
            return serviceBuses.GetOrAdd(queueOrTopicName, (queueOrTopicName) =>  serviceBusClient.CreateSender(queueOrTopicName) );
        }
    }

mjkent on Oct 21, 2021

If someone else stumbles upon this, we found a few ways to fix this:

Switch to Windows App Service
Switch to NAT Gateway
Upgrade to Net 6.0

🤷

mjkent on Feb 27, 2022

Inside CoreServiceRegistrator.Register(services);: services.AddTransient<ITransientClass, TransientClass>();

mjkent on Oct 25, 2021

Sounds good - btw you can actually just do this:

var queueMessage = new ServiceBusMessage(message);

instead of this:

var queueMessage = new ServiceBusMessage(Encoding.UTF8.GetBytes(message));

The SDK will do the UTF-8 conversion for you so there is no change perf or functionality-wise but I just wanted to point out that a constructor that takes a string is available.

JoshLove-msft on Oct 21, 2021