runtime: IOException on '/etc/ssl/certs' only on mcr.microsoft.com/dotnet/core/aspnet:3.1-alpine image
We’re running into a possible race condition when establishing SSL connections from .NET Core 3.1-alpine based docker images. We’ve not seen it happening before when using Debian based images.
It only happens with a very small amount of calls (this one occurred in a MassTransit consumer, but we’ve also seen it happen in controller actions, and establishing MongoDB connections).
Because we only see it happening on the 3.1-alpine image, we think it may be image-related.
A log of what happens:
MT-Fault-ConsumerType: MyApp.Api.Consumers.ManifestAvailableConsumer
MT-Fault-ExceptionType: System.IO.IOException
MT-Fault-Message: The process cannot access the file '/etc/ssl/certs' because it is being used by another process.
MT-Fault-MessageType: .Messaging.VideoUrl.Events.ManifestAvailable
MT-Fault-StackTrace: at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry(Byte* entryBufferPtr, Int32 bufferLength)
at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
at System.Linq.Enumerable.AppendPrependIterator`1.LoadFromEnumerator()
at System.Linq.Enumerable.AppendPrepend1Iterator`1.MoveNext()
at Internal.Cryptography.Pal.CachedSystemStoreProvider.LoadMachineStores(DirectoryInfo rootStorePath, FileInfo rootStoreFile)
at Internal.Cryptography.Pal.CachedSystemStoreProvider.GetCollections()
at Internal.Cryptography.Pal.CachedSystemStoreProvider.GetNativeCollections(SafeX509StackHandle& root, SafeX509StackHandle& intermediate)
at Internal.Cryptography.Pal.OpenSslX509ChainProcessor.InitiateChain(SafeX509Handle leafHandle, DateTime verificationTime, TimeSpan remainingDownloadTime)
at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
at System.Net.Security.CertificateValidation.BuildChainAndVerifyProperties(X509Chain chain, X509Certificate2 remoteCertificate, Boolean checkCertName, String hostName)
at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertValidationCallback remoteCertValidationCallback, ProtocolToken& alertToken)
at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
MT-Fault-Timestamp: 2020-07-27T10:58:37.6888414Z
MT-Host-Assembly: MyApp.Api
MT-Host-AssemblyVersion: 1.0.0.0
MT-Host-FrameworkVersion: 3.1.6
MT-Host-MachineName: placement-6d49dd57d9-mq45j
MT-Host-MassTransitVersion: 6.2.0.0
MT-Host-OperatingSystemVersion: Unix 4.15.0.1082
MT-Host-ProcessId: 1
MT-Host-ProcessName: dotnet
MT-Reason: fault
dtdTraceTagInfo: FW4;-111872945;5;-1869341797;401980;0;-1641347138;256;510e;2h01;3h90941b9b;4h06223c;5h01
publishId: 44
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 5
- Comments: 33 (19 by maintainers)
Do you have repro @JakobGSvendsen? AFAIK progress was not made as we cannot reproduce it.
bump
still experiencing the same issue with mcr.microsoft.com/dotnet/aspnet:6.0-alpine , downgrading to mcr.microsoft.com/dotnet/aspnet:5.0-alpine : \
MUSL 1.2.2 claims to have resolved CVE-2020-28928 who claims:
The MUSL devs claim the
wcsnrtombsfunction may not affect many but it’s possible .NET makes use of it thus the problem.Untested myself yet but I believe taking the
alpineimage and performing an upgrade on themusllib to 1.2.2 or newer may resolved the problem.Bump. We are experiencing this issue a lot, daily on .net 7. we are using a image based on 7.0-alpine any tricks?
Curious if that solves it for you? We have the issue with all Alpine images, regardless of the dotnet version, hence we moved away from Alpine completely unfortunately.
Unfortunately musl 1.2.2 still exhibits the issue.
@jeffhandley yeah that’s my question too:)
We’ve just moved away from the Alpine image unfortunately, any other image (we are using buster-slim now) doesn’t seem to have this problem.