runtime: Invalid pointer address during network call

Description

We intermittently get a The system detected an invalid pointer address in attempting to use a pointer argument in a call. during a network call using latest .NET 5.0.7 SDK on Azure web app using .NET 5 / 32bit.

I cannot provide a snippet because I can only reproduce the error in production environment (under load, not happening in a deployment slot or in a less busy environment). The code that fails basically is a HttpWebRequest call to get the response (not async). So I need a running app to reproduce.

The code that fails is 99% like this: https://github.com/LordVeovis/xmlrpc/blob/master/Kveer.XmlRPC/XmlRpcClientProtocol.cs

Configuration

  • Compiled with .NET Core SDK 5.0.301
  • Running on Azure Web Application
  • .NET 5 / 32 bit (this seems important, unable to reproduce in 64bit)

Regression?

The same code was running properly last week using ASP.NET 4.8 and built with .NET Sdk 3.1.404 with a 32bit configuration.

Other information

Stack trace (IP 1.2.3.4 and domain our.server.com is rewritten):

System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: The system detected an invalid pointer address in attempting to use a pointer argument in a call. [::ffff:1.2.3.4]:8069
  ?, in void Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
  ?, in void Socket.Connect(EndPoint remoteEP) x 5
  ?, in Stream ConnectHelper.Connect(string host, int port, CancellationToken cancellationToken)
System.Net.Http.HttpRequestException: The system detected an invalid pointer address in attempting to use a pointer argument in a call. [::ffff:13.69.121.169]:8069 (our.server.dns:8069)
  ?, in Stream ConnectHelper.Connect(string host, int port, CancellationToken cancellationToken)
  ?, in ValueTask<Stream> HttpConnectionPool.ConnectToTcpHostAsync(string host, int port, HttpRequestMessage initialRequest, bool async, CancellationToken cancellationToken)
  ?, in async ValueTask<ValueTuple<Socket, Stream, TransportContext, HttpResponseMessage>> HttpConnectionPool.ConnectAsync(HttpRequestMessage request, bool async, CancellationToken cancellationToken)
  ?, in async ValueTask<ValueTuple<HttpConnection, HttpResponseMessage>> HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, bool async, CancellationToken cancellationToken)
  ?, in async ValueTask<(HttpConnectionBase connection, bool isNewConnection, HttpResponseMessage failureResponse)> HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, bool async, CancellationToken cancellationToken)
  ?, in async ValueTask<HttpResponseMessage> HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, bool async, bool doRequestAuth, CancellationToken cancellationToken)
  ?, in async ValueTask<HttpResponseMessage> RedirectHandler.SendAsync(HttpRequestMessage request, bool async, CancellationToken cancellationToken)
  ?, in HttpResponseMessage HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
  ?, in HttpResponseMessage SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
  ?, in HttpResponseMessage DelegatingHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
  ?, in async ValueTask<HttpResponseMessage> DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, bool async, CancellationToken cancellationToken)
  ?, in HttpResponseMessage DiagnosticsHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
  ?, in HttpResponseMessage HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
  ?, in HttpResponseMessage HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
  ?, in async ValueTask<HttpResponseMessage> HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, bool async, bool emitTelemetryStartStop, CancellationToken cancellationToken)
  ?, in HttpResponseMessage HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
  ?, in async Task<WebResponse> HttpWebRequest.SendRequest(bool async)
  ?, in WebResponse HttpWebRequest.GetResponse()
System.Net.WebException: The system detected an invalid pointer address in attempting to use a pointer argument in a call. [::ffff:1.2.3.4]:8069 (our.server.dns:8069)
  ?, in WebResponse HttpWebRequest.GetResponse()
  File "C:\**REDACTED**\XmlRpcClientProtocol.cs", line 810, col 4, in WebResponse XmlRpcClientProtocol.GetWebResponse(WebRequest request)
  File "C:\**REDACTED**\XmlRpcClientProtocol.cs", line 164, col 4, in object XmlRpcClientProtocol.Invoke(object clientObj, MethodInfo mi, params object[] parameters)
  ?, in object XmlRpcProxy8a6acf37-0e1b-4b8f-a9db-7d5a34e64031.login(in string databaseName, in string userName, in string password)
  File "C:\**REDACTED**\HealthChecks\OpenErpHealthCheck.cs", line 44, col 9, in Task<HealthCheckResult> OpenErpHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken)

I could also provide a Snapshot Dump from Azure AI but not in this channel.

I don’t have any clue about what the problem can be, I tried to use HttpClient instead of HttpWebRequest but the problem persisted (also, HttpWebRequest uses HttpClient internally).

The problem only occurs in production environment (I can imagine that is due to the load) and with a 32bit configuration but we were not able to reproduce in any other environment not even in deployment slots with 100% the same configuration as the production site.

The only workaround we found was switching to 64bit.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (11 by maintainers)

Most upvoted comments

I ran the command in the current machine:

[System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      14393  0

Not sure if that was the version in the moment the bug was happening, but the web app is exactly the same. I tested as well on another web app that also had problems and is the same version, but they are not the same plan, just to be sure.

@mcliment thank you for the dump!

I had a look, both IPEndPoint and SocketAddress seem to be valid and correct, despite WSAConnect failing with WSAEFAULT. The most sane thing I can think of is a bug in winsock DualMode sockets on 32 bit. I’m about to contact the winsock team if they are aware of any known issues. Do you know the Windows version ([System.Environment]::OSVersion.Version) of the Azure machine?

@mcliment first of all, apologies for the long delay.

The reported error message means we are getting a WSAEFAULT socket error from winsock connect, which might be an issue around IPV4 - IPV6 socket address mapping in the .NET Socket code, however it’s very hard to tell more without seeing a repro in your environment, or at least a dump. I would be also interested about details at the void Socket.Connect(EndPoint remoteEP) x 5 call stack line.

@karelz is on vacation now so can’t ask him if he got the email with the dump, if you are still interested sharing it, maybe we can discuss it on twitter (just followed you).

Couple of things you can try:

  1. Replace the host name with a string containing the IP4 address of the server in your application
  2. Replace the host name with a string containing the IPv6-mapped address ([::ffff:1.2.3.4])
  3. With .NET 6 preview 7 (should be available in about a week): Set the environment variable DOTNET_SYSTEM_NET_DISABLEIPV6 to 1 before running your application. (Or set the AppContext switch System.Net.DisableIPv6 to true.)

I’m curious if any of these workarounds help.