azure-functions-host: DefaultHttpProxyService does not take into account cancellation (ForwarderError.RequestCanceled)

Proxy not forwarding cancellation token correctly to function app

Canada Central example @pmerineau first reported https://github.com/Azure/azure-functions-dotnet-worker/issues/2417#issuecomment-2455793340

Sending InvocationCancel request for invocation: '0577cf46-de3f-47ba-9751-0750cc28ba18'
And your function invocation is failing, likely due to this proxy that is not forwarding the cancellation request to your app properly:
Exception while executing function: Functions.<YourFunctionName> ---> System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled. System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request

North Europe example @stuartf-aida first reported https://github.com/Azure/azure-functions-dotnet-worker/issues/2417#issuecomment-2455857336

Sending InvocationCancel request for invocation: 'a11b8045-59cf-4db2-90eb-356d561e2fb2'
Exception while executing function: Functions.<YourFunctionName> ---> System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled. System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request

As mentioned by @liliankasem

The error is actually happening on the Functions Host side in the RetryProxyHandler so I would suggest opening the issue there and I will continue that investigation there as well.

I hope this is what you need, Lilian!

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 17 (6 by maintainers)

Most upvoted comments

Providing a quick update on this

We can use this issue to track the Host bug with DefaultHttpProxyService We can use the https://github.com/Azure/azure-functions-dotnet-worker/issues/2417 to track fixing the ObjectDisposedException bug

The PRs fixing the two issue mentioned have been merged, but the host change will not be released till the end of February or potential early March.

It’s also worth noting that the fix going into the dotnet worker can be used right away by upgrading the nuget package for the AspNet Http extension (when it is published), so half the issue is fixed there (the HttpContext disposed issue). The second half of the problem (Failed to proxy request with ForwarderError: RequestCanceled.) will be fixed when the host is released.

This issue automatically closed when the PR went in, but I will keep it open until the host release with this fix is officially out to make sure this isn’t forgotten.

Okay so it looks like we have a fun two part bug.

Host

The first part is in the host. The DefaultHttpProxyService has this code which does not take into account cancellation (ForwarderError.RequestCanceled). Whether cancellation is handled or not by the worker/customer code, we will always get this error message when an invocation is cancelled:

System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled

https://github.com/Azure/azure-functions-host/blob/287ceacce69d2c5916a03cfda341023bf2a1fa3a/src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs#L55-L66

Worker

There is also a bug in the AspNetCore worker extension which breaks the cancellation behaviour. In the DefaultHttpCoordinator, we pass in the cancellation token when we create the ContextReference:

var contextRef = _contextReferenceList.GetOrAdd(invocationId, static id => new ContextReference(id));
contextRef.SetCancellationToken(context.CancellationToken);
contextRef.FunctionContextValueSource.SetResult(context);

code

As a result, when a function invocation is cancelled, we cancel that same cts that is being used by ContextReference:

cancellationTokenSource?.Cancel();

code

So, we end up with a disposed HttpContext when we try to handle the invocation result:

ActionContext actionContext = new ActionContext(httpContext, httpContext.GetRouteData(), new ActionDescriptor());
await actionResult.ExecuteResultAsync(actionContext);

code

This is where the ObjectDisposedExecpetion comes from:

Exception: System.ObjectDisposedException: Request has finished and HttpContext disposed.
Object name: 'HttpContext'.
   at Microsoft.AspNetCore.Http.DefaultHttpContext.ThrowContextDisposed()
   at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Features()
   at Microsoft.AspNetCore.Routing.RoutingHttpContextExtensions.GetRouteData(HttpContext httpContext)
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 54
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
Stack:    at Microsoft.AspNetCore.Http.DefaultHttpContext.ThrowContextDisposed()
   at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Features()
   at Microsoft.AspNetCore.Routing.RoutingHttpContextExtensions.GetRouteData(HttpContext httpContext)
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 54
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89

Next Steps

I’ll speak with the team and discuss how we want to handle these issues.

  • We can use this issue to track the Host bug with DefaultHttpProxyService
  • We can use the dotnet-isolated issue to track fixing the ObjectDisposedException bug

we are on the same version and the issue still persist

"version":"4.1038.300.25164","versionDetails":"4.1038.300+65affe45679ffcc0c3b9059bd9d56a875d955eb3","platformVersion":"103.0.7.286"

I can confirm that our instances updated to this version today as well. "version":"4.1038.300.25164" If you would like to check yours you can use this: https://github.com/Azure/azure-functions-host/wiki/How-do-I-check-the-functions-runtime-version%3F

Thanks

The fix is rolling out with host version 4.x38, I don’t have an accurate ETA but is currently in progress - maybe late/mid March?

Hi, any updates on ETA?

Please kindly restart the function, I tried and it updated to v4.1038.300.25164

The fix is rolling out with host version 4.x38, I don’t have an accurate ETA but is currently in progress - maybe late/mid March?

Hi, any updates on ETA?

Same issue here. Is there any update on a fix? Alternatively, could you please provide instructions on how to hide / downgrade those exceptions?

This may be considered as “noise” for your team but when you have a production environment with Application Insights alerts configured with a certain threshold of exceptions and support staff being alerted each time that threshold is met, this has a real and costly impact.

We will hold on for now and keep running Functions on .NET 6 until we can find a better solution.

Thank you

No ETA at the moment, but we are working on it.

However, there should not be any real impact as this is mostly noise/badly handled exception on our end. If your function invocation is getting cancelled it is likely due to a client disconnect or something else shutting things down. If you handle the cancellation token in your code, your invocation should still run till completion even if it fails.

If there is real production impact, please open a customer support issue via the Azure portal so we can investigate further. Thanks!

Hi I am facing this too. I have a function app on node.js 20.0 LTS stack. Basically, I developed multiple http trigger functions within a function app, tested them via postman, deployed it on azure function app. The moment I run any of them, I see the following error

Image

@liliankasem It’s dotnet isolated, v8. Http trigger binding.

We’ve shifted our code now to use an alternative, based on the workaround for the previous issue where I originally posted. That seems (so far) to have solved this, but it’s hard to say. We don’t (and didn’t) have a situation that reliably caused this to happen - most invocations would not error. We can use the same payload on the http trigger such that the same code runs and we’ve not successfully reproduced the issue.

The function doesn’t do very much.

It uses Azure.Security.KeyVault.Secrets (4.7.0) to grab a secret from the keyvault, Environment.GetEnvironmentVariable to get an environment variable and Azure.Storage.Queues (12.20.1) to put an entry on a queue

and along the way a bit of string manipulation

Thanks very much for your keen attention on these issues!