azure-functions-dotnet-worker: Azure function intermittent error during cold start after converting to .NET 8 isolated worker
What version of .NET does your existing project use?
.NET 6
What version of .NET are you attempting to target?
.NET 8
Description
I have an Http Trigger Azure function that has been working fine for the past years, It is an Azure Function running on a Windows Consumption App Service Plan, I recently updated from .NET6 in-process to .NET8 isolated worker model.
After the upgrade to isolated worker, I am now seeing intermittent errors when calling the function, the function seems to crash after a cold start and errors are written to AppInsights.
Calling the function a second time minutes later the call will succeed and then subsequent calls will generally work for a while. Then we will typically see another error after the next cold start - although not always - sometimes it works fine.
The error I receive is :
- Timed out waiting for the function start call. The operation has timed out.
I tried updating all the libraries to the latest version and increasing the function timeout to 10 minutes, but still the issue exist.
One thing that reduces the issue to 1 or 2 times per day is this variation of libraries versions :
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.3.0"/>
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />
I want to add that this happened to all Http-triggered Azure functions we have at different Azure function apps. and we never had these exceptions before migration.
Project configuration and dependencies
That’s my Program.cs file :
var host = new HostBuilder()
.ConfigureFunctionsWebApplication()
.ConfigureServices((context, services) =>
{
var configuration = new ConfigurationBuilder()
.AddEnvironmentVariables()
.Build();
var applicationInsightsConnectionString = "connectionString";
services.AddApplicationInsightsTelemetryWorkerService(options => options.ConnectionString = applicationInsightsConnectionString);
})
.Build();
host.Run();
host.js:
{
"version": "2.0",
"functionTimeout": "00:10:00",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
},
"enableLiveMetricsFilters": true
}
}
}
And Packages references:
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.CosmosDB" Version="4.9.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.3.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />
<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />
<PackageReference Include="System.Formats.Asn1" Version="8.0.1" />
<PackageReference Include="System.Text.Json" Version="8.0.4" />
</ItemGroup>
Link to a repository that reproduces the issue
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 7
- Comments: 75 (12 by maintainers)
Wanted to provide an update on this and thank you all for the patience.
The platform updates to address this issue are currently rolling out with version 4.1037 of the host.
We’ll keep this issue open until the deployment is complete.
Thank you again!
Thank you for all the reports and additional details! That has been really helpful and, as a result, the issue has been identified and a fix put in place in the host.
The issue has is caused by a Windows Consumption specific behavior in cases where initialization of the HTTP listener takes longer than expected.
I don’t have an immediate ETA to share at this point, but wanted to share the information above as it may help some apply a mitigation while waiting for that to rollout (e.g., optimizing the worker bootstrapping/ensuring the listener startup is not blocked by other calls).
Will keep this updated, but thanks again for the additional information shared.
@fabiocav regarding your answer is that right the release is not yet to be know when happen but you have a fix in the code?
Please tell us more and give some examples for “e.g., optimizing the worker bootstrapping/ensuring the listener startup is not blocked by other calls”. How such optimizing should look like , what are bad habit? The workaround is really vague for me what to look for in the existing code.
Ah got it! We are aware of this issue and are actively investigating.
Additional telemetry has been added to ensure we have detailed information when the failure occurs. Will keep this open as investigation is still ongoing.
Just to report here, my functions are based in North Europe region and when they were upgraded to 4.1037 , literally all the issues were gone. I had nothing to do apart from sit and wait and check that everything works which indeed after hundreds of thousands of executions (cold/not cold …) over dozens of different apps, I can confirm everything works perfectly.
Hi I think the issue is more fundamental and on the SDK side. I have a simple function which uses no external service thus should not be any delays in the startup and it’s also throwing a 500 error during cold start. This issue has been going on for more than 6 months now, really hoping for a fix to be released soon.
@fabiocav you mention it is windows consumption specific, but I am experiencing it on a linux consumption plan.
@pauld-hf the issue is being actively investigated with the broader team. We’ll provide updates here as we make progress, but in the meantime, if anyone reporting this could share the app name (or the an invocation ID) with a time range when the issue was observed, that would be very helpful.
Thank you!
We are also experiencing problems with this since moving to .NET 8 isolated function which was recommended to us by Microsoft.
Could there please be an update from MS (@fabiocav @aishwaryabh) as to if this is actively being looked at or maybe some guidance as to how to mitigate the problem?
For our functions that we are seeing this behaviour we are running in consumption tier, the all nuget packages at the time of this message are the latest versions, the function apps are configured to run as 64bit on windows, the host is starting in
program.csusinghost.RunAsyncLooking in app insights at the failed 500 requests for this problem we see the following:
A
System.TimeoutExceptionfollowed by twoSystem.Net.Sockets.SocketExceptionsThe
System.Net.Sockets.SocketExceptionhave the following message:Exception while executing function: Functions.OurFunction Failed to proxy request with ForwarderError: Request An attempt was made to access a socket in a way forbidden by its access permissions. (localhost:51114) An attempt was made to access a socket in a way forbidden by its access permissions.With a stack trace of which can been seen below.
Subsequent calls to the function work.
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108) Inner exception System.InvalidOperationException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw: at Microsoft.Azure.WebJobs.Script.Grpc.DefaultHttpProxyService+<EnsureSuccessfulForwardingAsync>d__6.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: //src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs:65) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Grpc.GrpcWorkerChannel+<InvokeResponse>d__103.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: //src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs:1102) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: //src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:101) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: //src/WebJobs.Script/Description/FunctionInvokerBase.cs:82) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator+<Coerce>d__3`1.MoveNext (Microsoft.Azure.WebJobs.Script, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: //src/WebJobs.Script/Description/FunctionGenerator.cs:225) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:53) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306) Inner exception System.Net.Http.HttpRequestException handled at Microsoft.Azure.WebJobs.Script.Grpc.DefaultHttpProxyService+<EnsureSuccessfulForwardingAsync>d__6.MoveNext: at System.Net.Http.HttpConnectionPool+<ConnectToTcpHostAsync>d__98.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__97.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__99.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Http.HttpConnectionPool+<AddHttp11ConnectionAsync>d__74.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1+<WaitWithCancellationAsync>d__1.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Http.HttpConnectionPool+<GetHttp11ConnectionAsync>d__76.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Http.HttpConnectionPool+<SendWithVersionDetectionAndRetryAsync>d__84.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Http.DiagnosticsHandler+<SendAsyncCore>d__8.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Microsoft.Azure.WebJobs.Script.Grpc.RetryProxyHandler+<SendAsync>d__4.MoveNext (Microsoft.Azure.WebJobs.Script.Grpc, Version=4.34.0.0, Culture=neutral, PublicKeyToken=null: //src/WebJobs.Script.Grpc/Server/RetryProxyHandler.cs:33) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at Yarp.ReverseProxy.Forwarder.HttpForwarder+<SendAsync>d__8.MoveNext (Yarp.ReverseProxy, Version=2.0.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) Inner exception System.Net.Sockets.SocketException handled at System.Net.Http.HttpConnectionPool+<ConnectToTcpHostAsync>d__98.MoveNext: at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException (System.Net.Sockets, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult (System.Net.Sockets, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Sockets.Socket+<<ConnectAsync>g__WaitForConnectWithCancellation|277_0>d.MoveNext (System.Net.Sockets, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e) at System.Net.Http.HttpConnectionPool+<ConnectToTcpHostAsync>d__98.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
Any updates @fabiocav ? We’re having the same issues.
@fabiocav I’m on 4.1037.0.23521 and still hitting it.
Invocation IDs: 1b5b68b2-ed4c-426c-8c33-b9f33080f0e0 cd0646e8-c8e4-46df-a455-a18a71081685
These are from today and are in WestUS.
Cleared up here as well, thanks all!
Any findings/news about this would be appreciated. We also migrated many function apps from .NET 6 in-process to .NET8 isolated and are getting this error from time to time. Seen it pop up in premium plan also but usually it is in consumption plan functions.
Same here, these are back: Timed out waiting for the function start call. Invocation: ‘5527f60c-2c36-475a-82b7-eaaacc2c4a91’. The operation has timed out. On SDK version azurefunctions: 4.1039.400.25216
Very sad that the error comes back as we also have a lot of 429 problems and had to pin the functions to 4.1037 to make sure they run.
Seeing the same behavior as well on runtime version 4.1039.400.25216 (and already running on 64 bit). Lots of 429’s happening, already opened a support case but not resolution yet.
The only way I got this working was by replacing “new HostBuilder().ConfigureFunctionsWebApplication()” with “new HostBuilder().ConfigureFunctionsWorkerDefaults()” but that comes with the price of rolling up your sleeves and changing all the input/output parameters to start using HttpResponseData / HttpRequestData (Unit tests included!). Hard to justify to business people the amount of hours needed for swapping hundreds of endpoints because of some weirdness on the serverless infrastructure. Fingers crossed this nightmare is over soon
@fabiocav We rolled back to .net 6 in-process a couple of days ago and everything works nicely, not a single error in the financial flows.
We also tested the same with .net 9 isolated in our test environment but unfortunately it has the same issue as .net 8 isolated. So right now .net 8 and 9 are a showstopper for us, and we have many microservices we would like to upgrade but that is now blocked by this issue.
A side note: when I analyzed the errors on the function app "Diagnose and solve problems " we saw that is was not the cold start of the first instance that has the problems, it is actually when it scales out that we get the TimeoutException.
@dankrz I am sorry, I have no knowledge of if Flex Consumption plan would fix it.
@fabiocav an update would be appreciated (your last reply (above) is 3 weeks old)
Thanks
Providing an update. We’re working with our release groups and hope to have an ETA on that soon.
@dankrz given the root of the issue, ensuring you’re not making expensive calls in service initializations (i.e., hosted services, loggers or configuration providers that go make calls to hydrate data, etc.). I do realize this may not be feasible in all cases.
@fldsblzs / @puravu001 if you can share your application information, I can help confirm the root cause (sharing an invocation ID, region and timestamp of that invocation would be sufficient).
Thank you.
I’m hitting this on a Windows Premium Elastic plan.
We are also using a Linux plan with elastic premium instances.
In our case it is a consistent issue whenever the function tries to scale out when a request spike happens (from 2 always ready instances to ~20 instances). This happens nearly on a daily basis on our production environment -and also able to reproduce it by running load tests.
FWIW adding some further Invocation IDs and Timestamps from the most recent time when this occurred for us
6e9c3db9-754f-4656-b27d-7ebc3b882cea76609e7e-abca-4e37-810f-d0d96fcb9535Will look out for any updates - thanks for your help!
This issue seems like it’s it been kicked into the long grass from a pov of a consumer. Would it be possible to have an update i.e. is anyone looking at it, has it merited any investigation at all or made it to the bug list/ prioritised for fixing?
Is there any advice on what we should do? Do we convert to in-process which appears against MS recommendation post .NET6
We are experiencing the same issue. An update on mitigation and some sort of path to resolution is needed.
This variation of libraries versions has reduced the issue a lot per day, but it still exists. Not sure if it helps, but here it is :