azure-functions-host: DefaultHttpProxyService does not take into account cancellation (ForwarderError.RequestCanceled)
Proxy not forwarding cancellation token correctly to function app
Canada Central example @pmerineau first reported https://github.com/Azure/azure-functions-dotnet-worker/issues/2417#issuecomment-2455793340
Sending InvocationCancel request for invocation: '0577cf46-de3f-47ba-9751-0750cc28ba18'
And your function invocation is failing, likely due to this proxy that is not forwarding the cancellation request to your app properly:
Exception while executing function: Functions.<YourFunctionName> ---> System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled. System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request
North Europe example @stuartf-aida first reported https://github.com/Azure/azure-functions-dotnet-worker/issues/2417#issuecomment-2455857336
Sending InvocationCancel request for invocation: 'a11b8045-59cf-4db2-90eb-356d561e2fb2'
Exception while executing function: Functions.<YourFunctionName> ---> System.InvalidOperationException : Failed to proxy request with ForwarderError: RequestCanceled. System.IO.IOException : Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request
As mentioned by @liliankasem
The error is actually happening on the Functions Host side in the RetryProxyHandler so I would suggest opening the issue there and I will continue that investigation there as well.
I hope this is what you need, Lilian!
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 17 (6 by maintainers)
Providing a quick update on this
The PRs fixing the two issue mentioned have been merged, but the host change will not be released till the end of February or potential early March.
It’s also worth noting that the fix going into the dotnet worker can be used right away by upgrading the nuget package for the AspNet Http extension (when it is published), so half the issue is fixed there (the HttpContext disposed issue). The second half of the problem (Failed to proxy request with ForwarderError: RequestCanceled.) will be fixed when the host is released.
This issue automatically closed when the PR went in, but I will keep it open until the host release with this fix is officially out to make sure this isn’t forgotten.
Okay so it looks like we have a fun two part bug.
Host
The first part is in the host. The
DefaultHttpProxyServicehas this code which does not take into account cancellation (ForwarderError.RequestCanceled). Whether cancellation is handled or not by the worker/customer code, we will always get this error message when an invocation is cancelled:https://github.com/Azure/azure-functions-host/blob/287ceacce69d2c5916a03cfda341023bf2a1fa3a/src/WebJobs.Script.Grpc/Server/DefaultHttpProxyService.cs#L55-L66
Worker
There is also a bug in the AspNetCore worker extension which breaks the cancellation behaviour. In the
DefaultHttpCoordinator, we pass in the cancellation token when we create the ContextReference:code
As a result, when a function invocation is cancelled, we cancel that same cts that is being used by ContextReference:
code
So, we end up with a disposed HttpContext when we try to handle the invocation result:
code
This is where the ObjectDisposedExecpetion comes from:
Next Steps
I’ll speak with the team and discuss how we want to handle these issues.
DefaultHttpProxyServiceObjectDisposedExceptionbugwe are on the same version and the issue still persist
"version":"4.1038.300.25164","versionDetails":"4.1038.300+65affe45679ffcc0c3b9059bd9d56a875d955eb3","platformVersion":"103.0.7.286"I can confirm that our instances updated to this version today as well.
"version":"4.1038.300.25164"If you would like to check yours you can use this: https://github.com/Azure/azure-functions-host/wiki/How-do-I-check-the-functions-runtime-version%3FThanks
Please kindly restart the function, I tried and it updated to v4.1038.300.25164
Hi, any updates on ETA?
Same issue here. Is there any update on a fix? Alternatively, could you please provide instructions on how to hide / downgrade those exceptions?
This may be considered as “noise” for your team but when you have a production environment with Application Insights alerts configured with a certain threshold of exceptions and support staff being alerted each time that threshold is met, this has a real and costly impact.
We will hold on for now and keep running Functions on .NET 6 until we can find a better solution.
Thank you
No ETA at the moment, but we are working on it.
However, there should not be any real impact as this is mostly noise/badly handled exception on our end. If your function invocation is getting cancelled it is likely due to a client disconnect or something else shutting things down. If you handle the cancellation token in your code, your invocation should still run till completion even if it fails.
If there is real production impact, please open a customer support issue via the Azure portal so we can investigate further. Thanks!
Hi I am facing this too. I have a function app on node.js 20.0 LTS stack. Basically, I developed multiple http trigger functions within a function app, tested them via postman, deployed it on azure function app. The moment I run any of them, I see the following error
@liliankasem It’s dotnet isolated, v8. Http trigger binding.
We’ve shifted our code now to use an alternative, based on the workaround for the previous issue where I originally posted. That seems (so far) to have solved this, but it’s hard to say. We don’t (and didn’t) have a situation that reliably caused this to happen - most invocations would not error. We can use the same payload on the http trigger such that the same code runs and we’ve not successfully reproduced the issue.
The function doesn’t do very much.
It uses Azure.Security.KeyVault.Secrets (4.7.0) to grab a secret from the keyvault, Environment.GetEnvironmentVariable to get an environment variable and Azure.Storage.Queues (12.20.1) to put an entry on a queue
and along the way a bit of string manipulation
Thanks very much for your keen attention on these issues!