MassTransit: Cannot develop locally using Localstack

Is this a bug report?

Yes.

Can you also reproduce the problem with the latest version?

Yes, I can. Here is a link to a sample repo. https://github.com/rabberbock/MassTransitLocalstack/

Environment

Operating system: Windows 10
Visual Studio version: Visual Studio 2019
Dotnet version: .Net Core 3.0 preview 5

Steps to Reproduce

Download the Sample project from https://github.com/rabberbock/MassTransitLocalstack.
Make sure you have .Net Core 3 preview 5 installed
Have localstack running on your computer. (I ran it via docker and added docker.localhost to the hosts file). I am using the sqs transport. I tried it with go-aws as well and got the same error.
Run the application, and notice the error when trying to execute the handler.

Expected Behavior

It should execute the handler with no errors.

Actual Behavior

Currently message deserialization is broken using localstack to emulate sqs and sns. Running this with a real AWS sqs and sns actually works.

 R-FAULT amazonsqs://us-east-1/my_queue 7db85235-948f-4060-b37f-9568e3e51de3 Value cannot be null.
      Parameter name: source
System.Runtime.Serialization.SerializationException: An exception occurred while deserializing the message envelope ---> System.ArgumentNullException: Value cannot be null.
Parameter name: source
   at System.Linq.ThrowHelper.ThrowArgumentNullException(ExceptionArgument argument)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at MassTransit.Serialization.JsonConsumeContext..ctor(JsonSerializer deserializer, IObjectTypeDeserializer objectTypeDeserializer, ReceiveContext receiveContext, MessageEnvelope envelope)
   at MassTransit.Serialization.JsonMessageDeserializer.MassTransit.IMessageDeserializer.Deserialize(ReceiveContext receiveContext)
   --- End of inner exception stack trace ---
   at MassTransit.Serialization.JsonMessageDeserializer.MassTransit.IMessageDeserializer.Deserialize(ReceiveContext receiveContext)
   at MassTransit.Serialization.SupportedMessageDeserializers.Deserialize(ReceiveContext receiveContext)
   at MassTransit.Pipeline.Filters.DeserializeFilter.Send(ReceiveContext context, IPipe`1 next)
   at GreenPipes.Filters.RescueFilter`2.GreenPipes.IFilter<TContext>.Send(TContext context, IPipe`1 next)
dbug: MassTransit.Messages[0]
      SEND amazonsqs://us-east-1/MassTransit-ReceiveFault 47360000-5d56-0015-5d31-08d6f2863ce7 MassTransit.ReceiveFault
dbug: MassTransit.Messages[0]
      SEND amazonsqs://us-east-1/MassTransit-ReceiveFault 47360000-5d56-0015-3386-08d6f2863ceb MassTransit.ReceiveFault

Reproducible Demo

https://github.com/rabberbock/MassTransitLocalstack.

Thanks for all your help!

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 24 (19 by maintainers)

Most upvoted comments

You can check the options used in the unit tests via docker-compose for localstack that I use to test MassTransit.

The test harness settings shown here are the key to connecting locally.

phatboyg on Mar 23, 2021

Thanks so much. Following with my example, this works fine to connect to localstack as per previous message.

var region = "localhost:4566";
var accessKey = "test";
var secretKey = "test";
var busControl = Bus.Factory.CreateUsingAmazonSqs(c =>
{
    var hostAddress = new Uri($"amazonsqs://{region}")
    var hostConfigurator = new AmazonSqsHostConfigurator(hostAddress);
    hostConfigurator.AccessKey(accessKey);
    hostConfigurator.SecretKey(secretKey);
    hostConfigurator.Config(new AmazonSimpleNotificationServiceConfig {ServiceURL = $"http://{region}"});
    hostConfigurator.Config(new AmazonSQSConfig {ServiceURL = $"http://{region}"});
    c.Host(hostConfigurator.Settings);
});

Amazing help, as always!

PS: Careful with localstack versions. Some of them don’t work. See https://stackoverflow.com/questions/66767011/localstack-with-masstransit-not-getting-messages

diegosasw on Mar 23, 2021

@gertjvr @phatboyg I got a chance to look into this some more.

So it is indeed a problem related to RawDelivery. The subscription does not have raw delivery set properly.

The example JSON on this page, https://aws.amazon.com/blogs/compute/managing-amazon-sns-subscription-attributes-with-aws-cloudformation/, uses "RawMessageDelivery": "true" to set the RawMessageDelivery. The screenshot attached shows that the RawDelivery in C# code is "RawMessageDelivery": "True". On a real AWS setup this does not seem to make a difference, but on localstack it does.

What are your thoughts on changing https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit.AmazonSqsTransport/Topology/Entities/TopicEntity.cs#L61 to TopicSubscriptionAttributes["RawMessageDelivery"] = "true";. If you agree to this idea I can open up a pull request with a couple of tests.

I tested it out on localstack with the fix and it did work.

rabberbock on Aug 27, 2019