azure-sdk-for-java: [BUG][azure-spring-boot-starter-servicebus-jms][qpid] Message is put to DLQ after MaxDeliveryCountExceeded even when message is processed without error/exception

This issue is related to azure support tickets TrackingID#2109220050000347 and Ticket-number #11091792 detailed logs are attached in first ticket.

Describe the bug We are consuming messages from servicebus (SB). While we don’t see the spikes in SB everything works well. However when there’s spike (a lot of incoming messages msgcount > 10k) messages are being retried and then put to DLQ with reason MaxDeliveryCountExceeded. It seems like message commit is not processed correctly in servicebus and/or the commit is not delivered to SB and we don’t see any error in client library.

Exception or Stack Trace no stacktrace, no exception

To Reproduce Steps to reproduce the behavior: that’s a question to you, but probably put > 10k messages to SB and try to process them.

Code Snippet Here in archivingUsageTrackingService we are sending message to another topic.

  @JmsListener(destination = "${azure.servicebus.queues.usageActivity}", containerFactory = JMS_FACTORY_NAME)
  public void consume(UsageMessage usageMessage) {
    log.info("UsageMessage received: {}", usageMessage);
    usageTrackingService.processUsageMessage(usageMessage);
    archivingUsageTrackingService.trackErsUsage(usageMessage);
  }

Following is the config for queue/topic factories

  @Bean
  public JmsListenerContainerFactory<?> fiscalJmsFactory(ConnectionFactory connectionFactory,
      DefaultJmsListenerContainerFactoryConfigurer configurer) {
    if (connectionFactory instanceof CachingConnectionFactory) {
      CachingConnectionFactory cachingConnectionFactory = (CachingConnectionFactory) connectionFactory;
      cachingConnectionFactory.setCacheProducers(false);
    }
    DefaultJmsListenerContainerFactory factory = new DfDefaultJmsListenerContainerFactory();
    factory.setErrorHandler(t -> log.error("Unable to process JMS message", t));
    factory.setExceptionListener(t -> log.error("Unable to process JMS message", t));
    factory.setMessageConverter(jacksonJmsMessageConverter());
    configurer.configure(factory, connectionFactory);
    return factory;
  }

  @Bean
  @ConditionalOnProperty(prefix = "df.azure.servicebus", name = "enableJmsTopic", havingValue = "true")
  public JmsListenerContainerFactory<?> fiscalJmsTopicFactory(ConnectionFactory connectionFactory,
      DefaultJmsListenerContainerFactoryConfigurer configurer) {
    if (connectionFactory instanceof CachingConnectionFactory) {
      CachingConnectionFactory cachingConnectionFactory = (CachingConnectionFactory) connectionFactory;
      cachingConnectionFactory.setCacheProducers(false);
    }
    DefaultJmsListenerContainerFactory factory = new DfDefaultJmsListenerContainerFactory();
    factory.setErrorHandler(t -> log.error("Unable to process JMS message", t));
    factory.setExceptionListener(t -> log.error("Unable to process JMS message", t));
    factory.setMessageConverter(jacksonJmsMessageConverter());
    factory.setSubscriptionDurable(true);
    configurer.configure(factory, connectionFactory);
    return factory;
  }

and this is our customized JmsListenerContainerFactory

@Getter
@Setter
public class DfDefaultJmsListenerContainerFactory extends DefaultJmsListenerContainerFactory {

  private Boolean exposeListenerSession = Boolean.FALSE;

  @Override
  protected void initializeContainer(DefaultMessageListenerContainer container) {
    super.initializeContainer(container);
    if (this.exposeListenerSession != null) {
      container.setExposeListenerSession(this.exposeListenerSession);
    }
  }

}

Expected behavior Whenever message is processed without any error it should be removed from SB queue and not moved to DLQ. OR when message couldn’t be removed from queue because of any reason, we should see some error in client library.

Screenshots Spike: here we can see some client errors, but no error/exception is reported by underlying library.

Setup (please complete the following information):

• OS: kubernetes
• Library/Libraries: azure-spring-boot-starter-servicebus-jms 3.7.0
• Java version: 11
• App Server/Environment: tomcat 9.0.52
• Frameworks: spring-boot 2.5.4

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added