aspnetcore: Signalr Client can't communicate with server after reconnect

Is there an existing issue for this?

  • I have searched the existing issues

Describe the bug

We have a issue where in rare cases after a client has lost connection to the server and reconnects it suddenly can’t acknowledges messages from server and can’t send messages to the server. Even if you restart the Client it still can’t send or acknowledge messages. But if you restart the server all works correctly again.

Expected Behavior

After it reconnects it should be possible to communicate between client & server.

Steps To Reproduce

It hasn’t been possible to reproduce this issue yet. It only happens in rare cases.

Exceptions (if any)

2023-10-07T09:53:23.3200136Z [Warn]: Connection started reconnecting due to an error System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred. Please see the innerException, if present, for more details. —> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) — End of inner exception stack trace — at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) at System.Net.TlsStream.EndRead(IAsyncResult asyncResult) at System.Net.PooledStream.EndRead(IAsyncResult asyncResult) at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketConnectionStream.<ReadAsync>d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__19.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() at System.Net.WebSockets.WebSocketBase.ThrowIfConvertibleException(String methodName, Exception exception, CancellationToken cancellationToken, Boolean aborted) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport.<StartReceiving>d__22.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.GetReadAsyncResult() at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token) at System.Threading.Tasks.ValueTask1.get_Result() at Microsoft.AspNetCore.SignalR.Client.HubConnection.<ReceiveLoop>d__81.MoveNext() System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) — End of inner exception stack trace — at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) at System.Net.TlsStream.EndRead(IAsyncResult asyncResult) at System.Net.PooledStream.EndRead(IAsyncResult asyncResult) at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization) — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketConnectionStream.<ReadAsync>d__21.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__19.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

2023-10-09T08:16:20.8985093Z [Warn]: Connection started reconnecting due to an error System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred. Please see the innerException, if present, for more details. —> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) — End of inner exception stack trace — at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) at System.Net.TlsStream.EndRead(IAsyncResult asyncResult) at System.Net.PooledStream.EndRead(IAsyncResult asyncResult) at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketConnectionStream.<ReadAsync>d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__19.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() at System.Net.WebSockets.WebSocketBase.ThrowIfConvertibleException(String methodName, Exception exception, CancellationToken cancellationToken, Boolean aborted) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport.<StartReceiving>d__22.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.GetReadAsyncResult() at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token) at System.Threading.Tasks.ValueTask1.get_Result() at Microsoft.AspNetCore.SignalR.Client.HubConnection.<ReceiveLoop>d__81.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsyncCore>d__69.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsync>d__56.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnectionExtensions.<InvokeCoreAsync>d__431.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at AmphiSys.Signalr.Client.ClientController.<InternalSendMessageToServerAsync>d__22.MoveNext() System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) at System.Net.TlsStream.EndRead(IAsyncResult asyncResult) at System.Net.PooledStream.EndRead(IAsyncResult asyncResult) at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization) — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketConnectionStream.<ReadAsync>d__21.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__19.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) 2023-10-09T08:16:20.8985093Z [Error]: SendMessageToServerAsync exception System.Net.WebSockets.WebSocketException (0x80004005): An internal WebSocket error occurred. Please see the innerException, if present, for more details. —> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) — End of inner exception stack trace — at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) at System.Net.TlsStream.EndRead(IAsyncResult asyncResult) at System.Net.PooledStream.EndRead(IAsyncResult asyncResult) at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketConnectionStream.<ReadAsync>d__21.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__19.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() at System.Net.WebSockets.WebSocketBase.ThrowIfConvertibleException(String methodName, Exception exception, CancellationToken cancellationToken, Boolean aborted) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.Http.Connections.Client.Internal.WebSocketsTransport.<StartReceiving>d__22.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result) at System.IO.Pipelines.Pipe.GetReadAsyncResult() at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token) at System.Threading.Tasks.ValueTask1.get_Result() at Microsoft.AspNetCore.SignalR.Client.HubConnection.<ReceiveLoop>d__81.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsyncCore>d__69.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsync>d__56.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNetCore.SignalR.Client.HubConnectionExtensions.<InvokeCoreAsync>d__431.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at AmphiSys.Signalr.Client.ClientController.<InternalSendMessageToServerAsync>d__22.MoveNext() System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult) --- End of inner exception stack trace --- at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult) at System.Net.TlsStream.EndRead(IAsyncResult asyncResult) at System.Net.PooledStream.EndRead(IAsyncResult asyncResult) at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncTrimPromise1.Complete(TInstance thisRef, Func3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization) — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketConnectionStream.<ReadAsync>d__21.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at System.Net.WebSockets.WebSocketBase.WebSocketOperation.<Process>d__19.MoveNext() — End of stack trace from previous location where exception was thrown — at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.WebSockets.WebSocketBase.<ReceiveAsyncCore>d__45.MoveNext() System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult) at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

.NET Version

No response

Anything else?

We are using .net 7 (7.0.10) on the server. The clients are running WPF framework 4.7.2 using Microsoft.AspNetCore.SignalR.Client 7.0.2

About this issue

  • Original URL
  • State: closed
  • Created 9 months ago
  • Reactions: 2
  • Comments: 16 (7 by maintainers)

Most upvoted comments

I just noticed this code and comment: https://github.com/Bulgrak/SignalRNoReconnectDemo/blob/98e3720391ff07655eae1449654e7966fb2d2f64/src/SignalRNoReconnectDemoServer/Worker.cs#L52C1-L53

Pretty sure your issue is because you’re blocking the Hub.OnConnectedAsync method indefinitely if there is an active send to a client, because you hold onto the semaphore forever while trying to send the message. When you block Hub.OnConnectedAsync there are no messages being read and processed from the connection, so while the logs show the WebSocket getting bytes, SignalR isn’t processing the data because you’ve blocked the connection method.

While this investigation points to a bug in your code, it does make me want to add some additional logging in SignalR.