Ugh, that’s unfortunate. This is a trivial fix that we should do.

The behavior change you’re seeing is because we stopped using the static Socket.ConnectAsync, and started using the instance variant. The static variant will create a non-dualmode Socket for each address family, while the instance variant operates just on the your specific Socket.

https://github.com/dotnet/runtime/blob/f066972c7aa9f9147347623617e789e70aff3d3f/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs#L1285-L1300

The change will be pretty low risk. How common we think it is should dictate if it goes to servicing. I’m assuming Azure App Service users would be thankful, at least 😃.

According to our current (probably incomplete?) understanding, the root cause of the issue is that App Service currently does not support dual-stack sockets with VNET integration.

A workaround idea (specific to .NET 5) is to utilize ConnectCallback and enforce creating IPV4 sockets:

public HttpClient CreateWorkaroundClient()
{
    SocketsHttpHandler handler = new SocketsHttpHandler
    {
        ConnectCallback = IPv4ConnectAsync
    };
    return new HttpClient(handler);

    static async ValueTask<Stream> IPv4ConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
    {
        // By default, we create dual-mode sockets:
        // Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp);

        Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
        socket.NoDelay = true;

        try
        {
            await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false);
            return new NetworkStream(socket, ownsSocket: true);
        }
        catch
        {
            socket.Dispose();
            throw;
        }
    }
}

@jonsagara @davebronson @kwesseling and others watching this issue: can you try if this workaround helps?

We’ve got a confirmation from the Azure team regarding the fix:

The patch has been out since December 23rd – it contained two changes. One was the flag WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS (= 1 to enable, 0 to disable), which enables this new behavior, and the second was logic to detect if the app is running .NET5 in which case we’d enable this automatically (and this flag would be used only for disabling the behavior in case of regression). If customers are setting this flag manually to resolve themselves, it could be one of two things – either our .net5 detection logic is not working right for them, or it only started to work for them because by adding the setting they triggered a restart, which is necessary for the patch to take effect. Easy test is if they remove the app setting entirely and check if they are still mitigated.

According to our information, in their next release the switch will be enabled by default, and it will also contain a fix for Hybrid Connections, but we can not provide any ETA or promise on their behalf.

I’m closing this now, since the original issue is fixed, and I believe related problems are better tracked in separate issues.

Thanks again for the detailed reports, and all the amazing contribution we’ve seen here from the community!

I removed the WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS configuration setting, restarted the app, and confirmed that VNet requests still work.

I’m going to delete both of the test app services now.

Thank you @antonfirsov and everyone else who helped!

Thank you very much for closing the circle here @antonfirsov , and to everyone involved in addressing the issue. Thanks to @karelz too for managing this ticket. Cheers

It looks like Microsoft has a fix per this comment: https://github.com/MicrosoftDocs/azure-docs/issues/45991#issuecomment-755582909

Adding the setting WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS=1 resolved this problem for our application

Thanks for explanation. Azure App Service team realized they will need to do similar fix for Hybrid Connections as well (similar to the one for VPN). I don’t have answer from them yet on tracking status of “Is it fixed and deployed yet?” … will let you know when/if I know.

I think we have enough for servicing.

@antonfirsov can you make a PR for 5.0 once the 6.0 one is merged, and we can talk to shiproom.

I personally would love to see this in servicing release as we have so far identified three external libraries in use that hide the socket behavior and would all require changes in order for them to work in our app service to vNet scenario. We address our internal vNet servers/services via IP without using internal DNS.

We are looking into this with the help of Azure App Services team, but it’s too early to provide a plan or ETA. Will update as soon as we can.

Confirmed on my test site. I added the setting to the App Service configuration, restarted the site, and the following call no longer throws:

https://appservicevnettest50.azurewebsites.net/Home/VNetRequest

The Azure App Service incident number is 219225320 if anyone works with support to get update / more information.

We do not have ETA and we were explicitly asked not to discuss any dates by the team. I can assure you they treat is as high priority and are doing their best to unblock customers (based in our discussion, and follow up communications I see).

We discussed the root cause and options with Azure App Service last week. We concluded it is primarily problem in their environment (OS claims to support IPv6, but it does not) and that’s where it should be fixed first. There is no need to have fix/workaround in .NET 5.0 servicing for it. BTW: We considered bunch of workarounds, incl. short-term ones, to speed up time-to-resolution for customers. None of them was reasonable in the situation.

We also talked with AWS representative last week. We are waiting on root cause from their side to decide if such environments are more common and we should do a preventive fix in .NET 5.0 servicing, or if these were just 2 accidental one offs.

As a result, we do not expect to ship a servicing fix at this moment.

@antonfirsov given that this is an accidental regression in .NET 5 as @scalablecory points out above, can you please put up a fix in master for .NET 6? We can then discuss if we need to service .NET 5 or not.

@lahma note that hot fix in .NET 5 is not necessarily the only / the fastest option. Key thing is to find out if App Service is the only environment this will ever happen. If they push fix on their side, servicing .NET 5 may not be needed.

FWIW, re the service patch for .NET 5, our org would have hit this in the next month or so when we start moving some stuff to .NET 5 as we have pretty much this same configuration as the OP:

I have an ASP.NET Core application running in Azure App Service. It makes calls to Elasticsearch running on an Ubuntu Server 16.04 LTS Azure Virtual Machine, connected to the App Service via an Azure VNet.

I would revert to the old logic. Just copy the code from 3.1 – iirc it has a non-obvious optimization re: CancellationTokenRegistration allocation.

If we need a simple sample, we should define that directly in docs and not rely on the implementation.

I’d like to note, that we started deploying a netcore5 application to azure, January this month. And the auto detect logic from the Azure team does not work for us. So to anyone who finds this issue. You might have to toggle the magic setting (WEBSITE_VNET_SUPPORT_DUAL_STACK_SOCKETS) yourself. Or utilize the workaround httpclient mentioned by @antonfirsov earlier in this thread.

Be aware though that if you do use the SocketHttpClient you will run into this problem: https://github.com/dotnet/runtime/issues/31261 (TLDR, no AI Dependency logging on your http calls)

@SFinistrosa I would ask Azure App Service why it does not work for you. @ericsampson also no idea, I would wait for Azure App Service official resolution …

Now I’m curious, is this a temporary workaround that’s planned to be set to true permanently without users having to do anything?

Further, if you have a single App Service Plan which has both an ASP.NET application and a NodeJS Azure Functions application assigned to it, upgrading the ASP.NET application to .NET 5 irreversibly breaks your NodeJS Azure Functions app if it has a VNet configuration set.

You receive this error in the host logs: gRPC client connection info is missing or incorrect (–port is 0)

Disabling the VNet makes this issue go away, but obviously, we use VNet for a reason and can’t turn it off.

When I say irreversibly, I mean that downgrading the .NET app back to .NET Core 3.1 does not solve the Azure Functions issue. Deleting the entire App Service Plan and recreating it also does not solve the issue. You have to create a brand new App Service Plan with a different name, and be careful not to put .NET 5 on it.

@karelz Hybrid Connections are very similar to a VPN. The main difference is that there is no public IP exposed. Instead some services installed on-premises connects to an Azure Relay, which is “routing” the requests.

https://docs.microsoft.com/en-us/azure/app-service/app-service-hybrid-connections

This is the best short overview of its architecture that I know of. Hybrid Connections use the Azure Relay service. Someone from the Azure Messaging team could help with anything on the Azure server side. https://azure.microsoft.com/en-us/resources/videos/azure-service-bus-relay-overview/

Thanks @karelz , I understand. Is there any way to know when this work has been done? Some sort of Azure case ID or something that I could relay to Support when asking them for an update in X weeks. Cheers

@scalablecory I can confirm that the ConnectCallback workaround posted above by @antonfirsov resolved the issue for me. Here are some additional details that may be relevant:

• The connection to the service in AWS only fails via a split-tunnel VPN (paloalto/GlobalProtect), I have a separate full-tunnel VPN connection that works fine in .NET 5.0
• The connection via the split-tunnel does work in .NET Core 3.1
• The issue presented as a timeout rather than forbidden socket access

System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (<redacted>:443)
 ---> System.Net.Sockets.SocketException (10060): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)

Did you see this: https://github.com/MicrosoftDocs/azure-docs/issues/45991

I got a ticket with MS Support, and indeed the issue is that some client libraries don’t force IPv4 (as TcpClient) as the virtual NIC installed on the App Services workers (for regional VNet integration) don’t support IPv6.

@antonfirsov Here you go: https://feedback.azure.com/forums/223579-azure-portal/suggestions/41965039-azure-app-service-http-requests-to-azure-vnet-ip-a

Will do.