azure-signalr: Connection to Azure service drops and cannot automatically recover
Yesterday our web applications had connections to the Azure SignalR service dropped. We got several errors like this:
Connection "0a2f27dc-80e1-451e-aa4d-999d4bb950c6" to the service was dropped.
System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
at async Task<WebSocketReceiveResult> System.Net.WebSockets.WebSocketBase+WebSocketOperation.Process(Nullable<ArraySegment<byte>> buffer, CancellationToken cancellationToken)
at async Task<WebSocketReceiveResult> System.Net.WebSockets.WebSocketBase.ReceiveAsyncCore(ArraySegment<byte> buffer, CancellationToken cancellationToken)
at async Task Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport.StartReceiving(WebSocket socket)
at void System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at void System.IO.Pipelines.Pipe.GetReadResult(ref ReadResult result)
at ReadResult System.IO.Pipelines.Pipe.GetReadAsyncResult()
at async Task Microsoft.Azure.SignalR.ServiceConnectionBase.ProcessIncomingAsync()
It looked like one application instance dropped all connections and the other only dropped one.
Then we got several thousand of these errors:
Error while sending message to the service.
Microsoft.Azure.SignalR.Common.ServiceConnectionNotActiveException: The connection is not active, data cannot be sent to the service.
at async Task Microsoft.Azure.SignalR.ServiceConnectionBase.WriteAsync(ServiceMessage serviceMessage)
at async Task Microsoft.Azure.SignalR.ServiceConnection.ProcessOutgoingMessagesAsync(ServiceConnectionContext connection)
Then several thousand of these:
Failed writing message. Aborting connection.
System.InvalidOperationException: Writing is not allowed after writer was completed.
at void System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoWritingAllowed()
at Memory<byte> System.IO.Pipelines.Pipe.GetMemory(int sizeHint)
at Span<byte> System.IO.Pipelines.Pipe+DefaultPipeWriter.GetSpan(int sizeHint)
at ValueTask<FlushResult> System.IO.Pipelines.PipeWriter.WriteAsync(ReadOnlyMemory<byte> source, CancellationToken cancellationToken)
at async Task Microsoft.AspNetCore.SignalR.HubConnectionContext.TryWritePingSlowAsync()
This went on for over an hour until the web applications were manually restarted. Then the apps were able to re-connect and things were fine.
Versions: JS client: 1.1.0 Microsoft.AspNetCore.SignalR: 1.1.0 Microsoft.Azure.SignalR: 1.0.6
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 32 (15 by maintainers)
This seems to be happening twice a week for us, forcing us to restart our service that connects to Azure SignalR.
We’re currently using Microsoft.Azure.SignalR version 1.0.9. This is on a Standard Tier, 1 unit Azure SignalR instance.
This is our startup code:
@yosigolan Connection limits are client+server connections in total, that is 20.
@johnrutherford we are testing the new version, we will at you when we published.