orleans: Event Hub Stream Provider exception (when EH partition is moved ? )
We’ve started to see this exception in our logs and can’t understand what it actually means and why it manifested itself after a minor deployment
[2017-06-01 03:29:25.020 GMT 23 ERROR 103328 PullingAgentSystemTarget/FF/b7b682b4-BullclipEventHub 100.73.86.45:11111] !!!!!!!!!! Exception while reading from queue.
Exc level 0: Microsoft.ServiceBus.Messaging.MessagingCommunicationException: No connection handler was found for virtual host 'bullclip-production.servicebus.windows.net:15168'. Remote container id is 'a76f3594b2e34a51b3ba23d602e016d8'. TrackingId:1563685fced940d28dc201a13b5e8da5_B6_B6, SystemTracker:SharedConnectionListener, Timestamp:6/1/2017 3:29:25 AM
at Microsoft.ServiceBus.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)
at Microsoft.ServiceBus.Messaging.MessageReceiver.RetryReceiveEventDataAsyncResult.TryReceiveEnd(IAsyncResult r, IEnumerable`1& messages)
at Microsoft.ServiceBus.Messaging.MessageReceiver.EndTryReceiveEventData(IAsyncResult result, IEnumerable`1& messages)
at Microsoft.ServiceBus.Messaging.EventHubReceiver.<ReceiveAsync>b__56_1(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.ServiceBus.Providers.EventHubAdapterReceiver.<GetQueueMessagesAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.Streams.PersistentStreamPullingAgent.<ReadFromQueue>d__35.MoveNext()
and along with it goes this one sometimes:
Orleans.Streams.QueueCacheMissException: Item not found in cache. Requested: EventHubSequenceToken(EventHubOffset: 4302746296, SequenceNumber: 5470, EventIndex: 0), Low: [EventSequenceToken: SeqNum=5472, EventIndex=0], High: [EventSequenceToken: SeqNum=5472, EventIndex=0]
at Orleans.Providers.Streams.Common.PooledQueueCache`2.SetCursor(Cursor cursor, StreamSequenceToken sequenceToken)
at Orleans.ServiceBus.Providers.EventHubQueueCache`1.GetCursor(IStreamIdentity streamIdentity, StreamSequenceToken sequenceToken)
at Orleans.ServiceBus.Providers.EventHubAdapterReceiver.GetCacheCursor(IStreamIdentity streamIdentity, StreamSequenceToken token)
at Orleans.Streams.PersistentStreamPullingAgent.<DoHandshakeWithConsumer>d__31.MoveNext()
I found this SO question with the similar exception
Could it be related? We have 2 30 mins periods of these exceptions… Does stream implementation handle such partition moves and reconnections well ?
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Comments: 27 (27 by maintainers)
Confirmed with event hub team that this is a bug in event hub monitoring system.
I reproduced the doubling issue in our side. I still couldn’t find the cause in our side. But it seems that it is related to the fact we are using eventhub batch reading, based on this issue I found on GitHub: https://github.com/Azure/azure-webjobs-sdk/issues/890