azure-sdk-for-net: [BUG] Azure Search SDK does not seem to respect cancellation

Library name and version

Azure.Search.Documents 11.4.0

Describe the bug

We are passing along the CancellationToken properly to the search SDK, but it didn’t seem to respect the cancellation times. Our services are set to timeout the HTTP request from our browser to our backend service at 15 seconds. This terminates the HTTP connection and should cascade cancellations throughout the stack, including azure search. However, we found that it didn’t happen. In fact, some search and indexing requests took 1-5 minutes to timeout.

In our logs for these requests our cancellation tokens were canceled at the 15-second mark. However, we see the elapsed time was well beyond 15 seconds.

Expected behavior

When cancelling a search or indexing request, this cancellation goes to the search service

Actual behavior

System.AggregateException: Retry failed after 4 tries. Retry settings
 can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.)
 (The operation was cancelled because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout.) (The operation was canceled.) ---> System.Threading.Tasks.TaskCanceledException: The operation was cancelled
 because it exceeded the configured timeout of 0:01:40. Network timeout can be adjusted in ClientOptions.Retry.NetworkTimeout. ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from
 the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException (125): Operation canceled --- End of inner exception stack trace --- at void System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken
 cancellationToken) at async ValueTask<int> System.Net.Security.SslStream.EnsureFullTlsFrameAsync<TIOAdapter>(TIOAdapter adapter) at async ValueTask<int> System.Net.Security.SslStream.ReadAsyncInternal<TIOAdapter>(TIOAdapter adapter, Memory<byte> buffer) at
 async ValueTask System.Net.Http.HttpConnection.InitialFillAsync(bool async) at async Task<HttpResponseMessage> System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, bool async, CancellationToken cancellationToken) --- End of inner exception
 stack trace --- at void System.Net.Http.HttpClient.HandleFailure(Exception e, bool telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) at async Task<HttpResponseMessage>
 System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request)+Core(?) at async ValueTask Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, bool async) at async ValueTask Azure.Core.Pipeline.HttpPipelineTransportPolicy.ProcessAsync(HttpMessage
 message, ReadOnlyMemory<HttpPipelinePolicy> pipeline) at async ValueTask Azure.Core.Pipeline.RequestActivityPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async) at async ValueTask Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage
 message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async) --- End of inner exception stack trace --- at void Azure.Core.Pipeline.ResponseBodyPolicy.ThrowIfCancellationRequestedOrTimeout(CancellationToken originalToken, CancellationToken timeoutToken,
 Exception inner, TimeSpan timeout) at async ValueTask Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async) at async ValueTask Azure.Core.Pipeline.LoggingPolicy.ProcessAsync(HttpMessage
 message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async) at async ValueTask Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async) at async ValueTask Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage
 message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async) --- End of inner exception stack trace --- at async ValueTask Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory<HttpPipelinePolicy> pipeline, bool async) at async
 Task<Response<SearchResults<T>>> Azure.Search.Documents.SearchClient.SearchInternal<T>(SearchOptions options, string operationName, bool async, CancellationToken cancellationToken) x 2 at async Task<Response<SearchResults<T>>> Azure.Search.Documents.SearchClient.SearchAsync<T>(string
 searchText, SearchOptions options, CancellationToken cancellationToken)

Received a 503 response after 207 seconds (should have been canceled at 15 seconds…)

Reproduction Steps

Call cancel on a search sdk method

Environment

Windows .NET 6

About this issue

Original URL
State: open
Created a year ago
Comments: 17 (9 by maintainers)

Most upvoted comments

this is a .NET 6 timeframe version: https://github.com/dotnet/runtime/blob/2ac023c8e9df61fe4557212cbd13956042fb47c5/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs It does not take the cancellation token.

It does, it’s just more obscured that it does. That method: https://github.com/dotnet/runtime/blob/2ac023c8e9df61fe4557212cbd13956042fb47c5/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs#L826 takes a struct argument, a TIOAdapter, and its ReadAsync method is called: https://github.com/dotnet/runtime/blob/2ac023c8e9df61fe4557212cbd13956042fb47c5/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs#L844 That adapter’s ReadAsync method is: https://github.com/dotnet/runtime/blob/2ac023c8e9df61fe4557212cbd13956042fb47c5/src/libraries/System.Net.Security/src/System/Net/Security/ReadWriteAdapter.cs#L33-L34 which you can see does pass in the token that was given to its ctor. And you can see the adapter being constructed with a token in all the places one is created, e.g. https://github.com/dotnet/runtime/blob/2ac023c8e9df61fe4557212cbd13956042fb47c5/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.Implementation.cs#L203

stephentoub on Jan 19, 2023