orleans: NullReferenceException in deserialization when mixing serializers

Hi, I have a strange case.

I have an EventConsumerGrain that retrieves events from an event sourcing database and calls an IEventConsumer.

One of these consumers throws a NullReferenceException:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Squidex.Domain.Apps.Entities.Contents.Text.GrainTextIndexer.On(Envelope`1 event) in /src/src/Squidex.Domain.Apps.Entities/Contents/Text/GrainTextIndexer.cs:line 68
   at Squidex.Infrastructure.EventSourcing.Grains.EventConsumerGrain.DispatchConsumerAsync(Envelope`1 event) in /src/src/Squidex.Infrastructure/EventSourcing/Grains/EventConsumerGrain.cs:line 232
   at Squidex.Infrastructure.EventSourcing.Grains.EventConsumerGrain.<>c__DisplayClass10_0.<<OnEventAsync>b__0>d.MoveNext() in /src/src/Squidex.Infrastructure/EventSourcing/Grains/EventConsumerGrain.cs:line 83
--- End of stack trace from previous location where exception was thrown ---
   at Squidex.Infrastructure.EventSourcing.Grains.EventConsumerGrain.DoAndUpdateStateAsync(Func`1 action, String caller) in /src/src/Squidex.Infrastructure/EventSourcing/Grains/EventConsumerGrain.cs:line 165

The code where the exception is thrown:

https://github.com/Squidex/squidex/blob/master/src/Squidex.Domain.Apps.Entities/Contents/Text/GrainTextIndexer.cs#L68

As you can see there is only a reference to the grain. The content event cannot be null because of the pattern matching.

If I scale down the cluster to 1 it works, so I guess it has something to do with the remote call. But I do not receive details.

EDIT: I am not sure yet, but I would say that it only happens under load, e.g. when many requests per second are done (~100)

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 28 (28 by maintainers)

Most upvoted comments

@ReubenBond I have another observation. I made the mistake again and now I get this:

message: "Insufficient data present in buffer."
stackTrace: "   at Orleans.Serialization.BinaryTokenStreamReader2.ThrowInsufficientData()
   at Orleans.Serialization.BinaryTokenStreamReaderExtensinons.TryReadSimpleType[TReader](TReader this, Object& result, SerializationTokenType& token)
   at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
   at Orleans.Serialization.BuiltInTypes.DeserializeInvokeMethodRequest(Type expected, IDeserializationContext context)
   at Orleans.Serialization.SerializationManager.DeserializeInner[TContext,TReader](SerializationManager sm, Type expected, TContext context, TReader reader)
   at Orleans.Runtime.Messaging.MessageSerializer.OrleansSerializer`1.Deserialize(ReadOnlySequence`1 input, T& value)
   at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message)
   at Orleans.Runtime.Messaging.Connection.ProcessIncoming()"
type: "System.InvalidOperationException"

SebastianStehle on Aug 13, 2020