DotNetCore 2.2.6 on linux takes a long time to make connections and publish messages to AWS AMQ

General

We run dotnetcore in containers on our OpenShift cluster. Most recently they failed on all environments because they can no longer connect to AWS AMQ. It feels like it’s running against a timeout of something. This issue only occurs when running dotnetcore on linux.

My linux guest system is an ubuntu 18.04 ( see dotnet --info below ). It also occurs on RHEL 7 containers

I eliminated the network as culprit:

• I can connect from my windows machine, but it takes a long time from my linux guest OS in the same machine.
• My linux guest OS can connect and post messages to AWS AMQ from a java program ( fast ).
• I asked our network guy to allow outgoing network traffic from my machine to that specific instance.
• I used wireshark to monitor the network traffic from my PC to see if it connected somewhere else ( which it didn’t ).

I eliminated AWS AMQ as the culprit:

• The AWS logs show no sign of something wrong
• The same queue works in a java
• We set up a different queue from scratch and the issue still occurs.

Wireshark shows pauses of ~20seconds at certain intervals. It is consistently around 20seconds, so it does feel like a timeout of some sort… Wireshark shows dotnetcore connecting with TLS 1.0 connection in both windows & linux using the same cipher suite ( TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA ). When executing the same code on windows it can connect just fine. When trying to connect to the AWS AMQ in java it is able to connect fine as well.

The only remaining “culprit” I can think of is something going wrong in the dotnetcore runtime itself unless I am missing something.

Below you have a C# code sample ( taken from the amazon documentation ). Building and running on windows - it works perfect. Build and running on Linux - It takes forever to connect and post messages ( > 50s ).

Note: I just noticed there is a slight version difference between my linux & windows regarding dotnetcore. I will update it and try that as well and add it as output.

System Information

Host System: Windows 10 Guest System: Linux 18.04

Dotnet SDK was installed through https://dotnet.microsoft.com/download/linux-package-manager/ubuntu18-04/sdk-current

AWS AMQ Information

Instance Type: mq.t2.micro Deployment mode: single-instance broker Broker Engine: ActiveMQ Broker engine version: 5.15.9

Build command

dotnet build --force -c Release dotnet publish -c Release -o ../publish/exploded

Run command

dotnet AWSMQSend.dll $@

Application output on linux: ( notice gaps in timestamps )

9/9/19 11:54:56 AM :: Creating Factory
9/9/19 11:54:56 AM :: Factory Created
9/9/19 11:54:56 AM :: Connection...
9/9/19 11:54:56 AM :: Done
9/9/19 11:54:56 AM :: Creating Session.....
9/9/19 11:55:27 AM :: Done
9/9/19 11:55:27 AM :: Using destination: queue://queue_filip
9/9/19 11:55:57 AM :: Start the connection so that messages will be processed
9/9/19 11:55:57 AM :: Sending message....
9/9/19 11:56:17 AM :: Received message with ID:   ID:fluxbox-34467-637033634349736262-1:0:1:1:1
9/9/19 11:56:17 AM :: Received message with text: Hello World!

Application output on windows:

9-9-2019 12:01:39 :: Creating Factory
9-9-2019 12:01:39 :: Factory Created
9-9-2019 12:01:39 :: Connection...
9-9-2019 12:01:40 :: Done
9-9-2019 12:01:40 :: Creating Session.....
9-9-2019 12:01:40 :: Done
9-9-2019 12:01:40 :: Using destination: queue://queue_filip
9-9-2019 12:01:40 :: Start the connection so that messages will be processed
9-9-2019 12:01:40 :: Sending message....
9-9-2019 12:01:40 :: Received message with ID:   ID:BEZAV1CL13911-53543-637036270142492136-1:0:1:1:1
9-9-2019 12:01:40 :: Received message with text: Hello World!

Application Code C# code

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

using System;
using Apache.NMS;
using Apache.NMS.Util;

namespace Apache.NMS.ActiveMQ.Test
{
public class Program
{
    public static void Main(string[] args)
    {
        string host = "<your-host>.mq.eu-central-1.amazonaws.com";
        string port = "61617";
        string userName = "<queue-username>";
        string password = "<queue-password>";

        Uri connecturi = new Uri("activemq:ssl://" + host + ":" + port);

        Console.WriteLine( DateTime.Now + " :: Creating Factory" );

        // NOTE: ensure the nmsprovider-activemq.config file exists in the executable folder.
        IConnectionFactory factory = new NMSConnectionFactory(connecturi);

        Console.WriteLine( DateTime.Now + " :: Factory Created" );

		Console.WriteLine( DateTime.Now + " :: Connection...");
		IConnection connection = factory.CreateConnection( userName, password);
		Console.WriteLine( DateTime.Now + " :: Done");
		Console.WriteLine( DateTime.Now + " :: Creating Session.....");
		//The line below takes 20 seconds to complete!
		ISession session = connection.CreateSession( AcknowledgementMode.AutoAcknowledge );
		Console.WriteLine( DateTime.Now + " :: Done");

		IDestination destination = SessionUtil.GetDestination(session, "queue://queue_filip");
		Console.WriteLine( DateTime.Now + " :: Using destination: " + destination);

		// Create a consumer and producer
		using(IMessageConsumer consumer = session.CreateConsumer(destination))
		using(IMessageProducer producer = session.CreateProducer(destination))
		{
			// Start the connection so that messages will be processed.
			Console.WriteLine( DateTime.Now + " :: Start the connection so that messages will be processed" );
			connection.Start();
			producer.DeliveryMode = MsgDeliveryMode.Persistent;
				
			// Send a message
			ITextMessage request = session.CreateTextMessage("Hello World!");
			request.NMSCorrelationID = "abc";
			request.Properties["NMSXGroupID"] = "cheese";
			request.Properties["myHeader"] = "Cheddar";

			Console.WriteLine( DateTime.Now + " :: Sending message...." );

			producer.Send(request);

			// Consume a message
			ITextMessage message = consumer.Receive() as ITextMessage;
			if(message == null)
			{
				Console.WriteLine( DateTime.Now + " :: No message received!");
			}
			else
			{
				Console.WriteLine( DateTime.Now + " :: Received message with ID:   " + message.NMSMessageId);
				Console.WriteLine( DateTime.Now + " :: Received message with text: " + message.Text);
			}
		}
    }
}
}

dotnet --info [Linux]

$ dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   2.2.401
 Commit:    729b316c13

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.2.401/

Host (useful for support):
  Version: 2.2.6
  Commit:  7dac9b1b51

.NET Core SDKs installed:
  2.2.401 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

dotnet --info [Windows]

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.104
 Commit:    73f036d4ac

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.16299
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.104\

Host (useful for support):
  Version: 2.2.2
  Commit:  a4fd7b2c84

.NET Core SDKs installed:
  2.2.104 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Updates

Update: Execution on linux is fast against a local AMQ docker container, without SSL:

9/9/19 2:00:18 PM :: Creating Factory
9/9/19 2:00:18 PM :: Factory Created
9/9/19 2:00:18 PM :: Connection...
9/9/19 2:00:19 PM :: Done
9/9/19 2:00:19 PM :: Creating Session.....
9/9/19 2:00:19 PM :: Done
9/9/19 2:00:19 PM :: Using destination: queue://queue_filip
9/9/19 2:00:19 PM :: Start the connection so that messages will be processed
9/9/19 2:00:19 PM :: Sending message....
9/9/19 2:00:19 PM :: Received message with ID:   ID:fluxbox-44221-637036344189323303-1:0:1:1:1
9/9/19 2:00:19 PM :: Received message with text: Hello World!

Edit: Spelling