SqlClient: After SqlException application state is corrupted

Environment: OS - Windows Server 2012 R2 .NET Core: 2.0.5 SQL Server 2016

Exception:

System.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 0 - The wait operation timed out.) ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
   at bool System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, uint waitForMultipleObjectsTimeout, bool allowCreate, bool onlyOneCheckConnection, DbConnectionOptions userOptions, out DbConnectionInternal connection)
   at bool System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource<DbConnectionInternal> retry, DbConnectionOptions userOptions, out DbConnectionInternal connection)
   at bool System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource<DbConnectionInternal> retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, out DbConnectionInternal connection)
   at bool System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource<DbConnectionInternal> retry, DbConnectionOptions userOptions)
   at bool System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource<DbConnectionInternal> retry)
   at Task System.Data.SqlClient.SqlConnection.OpenAsync(CancellationToken cancellationToken)
   at async Task<byte[]> Microsoft.Extensions.Caching.SqlServer.DatabaseOperations.GetCacheItemAsync(string key, bool includeValue, CancellationToken token)
   at async Task Microsoft.Extensions.Caching.SqlServer.DatabaseOperations.RefreshCacheItemAsync(string key, CancellationToken token)
   at async Task Microsoft.Extensions.Caching.SqlServer.SqlServerCache.RefreshAsync(string key, CancellationToken token)
   at async Task Microsoft.AspNetCore.Session.DistributedSession.CommitAsync(CancellationToken cancellationToken)
   at async Task Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
ClientConnectionId:f8356379-665e-484b-9093-e8e99ae10a72
Error Number:258,State:0,Class:20

This issue has come up in production in two separate occasions this week and once the ASP.NET Core app enters into this state only restarting the app resolves the problem. A few points I want to share:

1- It seems like something in the Connection Pool is getting corrupted once this happens. Since a restart fixes it.

2- The error seems related to some SSL issue based on the exception details (Provider SSL). However, our connection strings do not specify the Encrypt attribute.

3- There are a few issues reported with the same exception here, but in Linux/Mac environments. In this case this is an all Windows environment.