SqlClient: A transport-level error has occurred when receiving results from the server. (provider: Session Provider, error: 19 - Physical connection is not usable)

Yes on connection open. It’s part of transient fault handling. 40613 shall occur only when server is not available as per Azure docs. Also we only retry when opening connection so if this occurs in other cases, customer apps should.

No, we never found a root cause. To solve the issue we added a custom retry execution policy to our database context:

 options.ExecutionStrategy(dependencies =>
 {
     return new ExtendedSqlAzureExecutionStrategy(dependencies, 3, TimeSpan.FromSeconds(5));
 });

public ExtendedSqlAzureExecutionStrategy(ExecutionStrategyDependencies dependencies, int maxRetryCount,
    TimeSpan maxRetryDelay, List<int> errorNumbersToAdd = null) : base(dependencies, maxRetryCount, maxRetryDelay,
    errorNumbersToAdd)
{
}

protected override bool ShouldRetryOn(Exception exception)
{
    return base.ShouldRetryOn(exception) || IsPhysicalConnectionNotUsableSqlException(exception);
}

private bool IsPhysicalConnectionNotUsableSqlException(Exception ex)
{
    if (ex is SqlException sqlException)
    {
        return sqlException.Message.Contains("Physical connection is not usable") || 
               (sqlException.InnerException is not null && sqlException.InnerException.Message.Contains("Physical connection is not usable"));
    }

    return false;
}

This solved it for us.

We couldn’t reproduce the above issue after implementing the AppContext switch change mentioned here https://github.com/microsoft/dotnet/issues/1263#issuecomment-718884828 in Web.config file https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/appcontextswitchoverrides-element#switches-in-aspnet-apps

That error above is not 40613 so I’m curious to know too how this error 19 relates to 40613? @rajagopal-p Is there an internal error captured above?

This might be the reason for these issues and they have suggested to implement a retry mechanism for error code 40613 and now we are not receiving this type of errors.

Regarding this issue, our driver does retry on error code 40613 (NetFx | NetCore) so Microsoft.Data.SqlClient should not fail. This error code also applies to Serverless DBs getting resumed, which we support retrying on. Can you confirm with our latest released version if that works for you? [Please allow higher timeout around 90 secs if server takes time to resume]

@rajagopal-p

Do you recommend to update SqlClient to Microsoft.Data.SqlClient to avoid similar issues in the future since the development focus for System.Data.SqlClient client has changed? We are planning to upgrade that in our next release.

Yes, always. System.Data.SqlClient may not receive these updates, so we always recommend moving sooner rather than later. Also we’re going to introduce new reliability APIs in next releases to be able to configure retry externally too, PR #693 shall bring more hands on capabilities to perform retry.

@rajagopal-p You should always anticipate transient errors from Azure, and implement something like EF Core does here: https://github.com/dotnet/efcore/blob/main/src/EFCore.SqlServer/Storage/Internal/SqlServerTransientExceptionDetector.cs

Hi @cheenamalhotra

Based on the response we have received from MS Azure DB Team, This issue was related to a recent Azure DB infrastructure migration activity. They have upgraded some of their clusters to newer hardware across all the regions and our database was a part of that migration. This might be the reason for these issues and they have suggested to implement a retry mechanism for error code 40613 and now we are not receiving this type of errors.

Do you recommend to update SqlClient to Microsoft.Data.SqlClient to avoid similar issues in the future since the development focus for System.Data.SqlClient client has changed? We are planning to upgrade that in our next release.

Thanks, Rajagopal