azure-functions-host: Random "The service is unavailable." and "Azure Functions runtime is unreachable" errors
We have been running a couple of Azure Functions in various subscriptions, and every once in a while (about once a week), the entire Azure Function goes down and reports “The service is unavailable.” when accessing the Function-app via HTTP and reports “Azure Functions runtime is unreachable” in the Azure Portal.
HTTP response:
Portal:
The issue appears randomly, without any changes in our end (no deployment, etc.) and also resolves randomly without any interaction from our side.
In the “Activity log” an error is written for the job “Sync Web Apps Function Triggers” with status “Failed”:
"statusCode": "BadRequest",
"statusMessage": "{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (ServiceUnavailable) from host runtime.\",\"Target\":null,\"Details\":[{\"Message\":\"Encountered an error (ServiceUnavailable) from host runtime.\"},{\"Code\":\"BadRequest\"},{\"ErrorEntity\":{\"Code\":\"BadRequest\",\"Message\":\"Encountered an error (ServiceUnavailable) from host runtime.\"}}],\"Innererror\":null}",
The issue seems quite similar to #8519, however we are running Linux.
This is causing quite some problems, since we are no longer able to provide reliable service to our end-users.
Investigative information
- Timestamp: Wednesday, 27 July 2022, 07:47, Coordinated Universal Time (UTC)
- Function App version: V4
- Function App name: Please use Invocation ID
- Function name(s) (as appropriate): All functions within the app are non-working
- Invocation ID: None of the functions work, but to find our function-app: 2022-07-27T08:05:13.773Z - e95ba16f-f2a2-420c-bcfd-4891d54198ae
- Region: West-Europe
Repro steps
None that we can find
Expected behavior
Always work
Known workarounds
None
Related information
Hosting Model: Consumption Plan OS: Linux Version: V4 Hosting Model In-Process Language: C#/dotnet6 Configuration:
- WEBSITE_RUN_FROM_PACKAGE points to a valid file on a storage account
- WEBSITE_MOUNT_ENABLED: 1
- WEBSITE_ENABLE_SYNC_UPDATE_SITE: false
- WEBSITE_CONTENTAZUREFILECONNECTIONSTRING: Valid keyvault reference toa a storage account
- SCM_DO_BUILD_DURING_DEPLOYMENT: false
- FUNCTIONS_WORKER_RUNTIME: dotnet-isolated
- FUNCTIONS_EXTENSION_VERSION: ~4
- AzureWebJobsDisableHomepage: true
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 124 (2 by maintainers)
Seems like they are working on an issue, but once again, it does not show up in the health status…
As mentioned in then original issue, Those values point to valid values.
Given the extreme randomness of both the issue appearing and resolving itself, I hardly doubt any setting on our end is responsible.
We have experienced the problem again in West Europe. We bit the bullet and moved the function to North Europe and that seems to help for now. So really seems to be a region issue like suggested earlier in the thread already. We made an Azure Support Issue again because its becoming kinda ridiculous how long this is taking Microsoft to solve.
If “Throwing money at the problem” counts as a workaround, then yes. This ticket is specifically about the Consumption Plan.
Yes this is also happening yet again for us - West Europe, consumption plan, linux. This service claims to be a SLA of 99.95% uptime, but we are getting nowhere close to that over the last few weeks.
Thank you for mentioning this. For us this is the problem as well. Within seconds after creating a System Assigned Managed Identity on a Linux Consumption Plan function in West Europe, it crashes with the below error:
The behavior is predictable and reproducable.
Also potentially useful context info: We’re trying to connect to an App Configuration instance via Managed Identity.
I also have had this problem, linux host consumption plan west europe, my problem was related to enabling managed identity on the function app. Disabling that made it work again.
Edit: was trying to connect to cosmosdb via managed identity, but target service seems irrelevant since wp-sm below had the same error trying to access a different service via managed identity.
Presumably this your app is also in West Europe on a Linux consumption plan @espenja?
What I find most troublesome of all of this is that the Azure status page has had a green tick against Azure Functions in West Europe throughout, even whilst the Azure support team is tweeting that they are investigating an issue.
It just seems a bit disingenuous. If you know there is a degradation of service, and you are investigating, own up to it on your status page (and honour your SLAs).
Same issues as those already reported here: unable to use (Error 503) and deploy to functions (on linux) with consumption plan.
This is ridiculous. Not only have we been experiencing the issue mentioned here, but also a bug causing the entire runtime to fail when running a custom handler in the Linux runtime - which Azure Support does not acknowledge as a bug, even though they can’t point out what’s wrong with our configuration.
The Functions dev team should really focus on reliability and observability of their service if they want to keep their customers.
We are planning to migrate to some other platform because it looks like Microsoft is just unable to keep this service running without significant issues and bugs all the time.
Same issue happened for me (azure functions on linux consumption plan in west-europe). My azure functions went down a couple times today & yesterday with the error “The service is unavailable.”. When checking my functions in the azure portal, I see the message “Azure Functions runtime is unreachable”.
This seems to have happened a couple times for me in the past weeks 😦
Same
It’s working again in my end. Had about 2-3 hours of downtime.
Also experiencing this right now deploying from Azure DevOps. ARM-templates ran just fine. Deployed to staging environment and the deploy fails with error:
##[error]Error: Failed to sync triggers for function app 'name-APP-prod-staging'. Error: BadRequest - Encountered an error (ServiceUnavailable) from host runtime. (CODE: 400)After checking the Azure Function App, all our Functions have disappeared in both slots and we are seeing error “Azure Functions runtime is unreachable” under the “Functions tab”. All our post processing is now down in production, once again.
This is the umpteenth time we’re facing this issue. Starting to get real fed up. Completely unreliable when it fails or not. We have made no changes to setup or configuration of the Azure Functions, only code changes.
same region West Europe
The SAS token for the
WEBSITE_RUN_FROM_PACKAGEurl was expired in my case, updating the url fixed the issue for me.Had this issue myself, just now. resolved itself but this is worries some. We depend on this to work for critical infrastructure. Should we look elsewhere?
I had the same issue, on azure function linux based on consumption plan on west europe zone. A request through postman get a “The service is unavailable.” We can’t deploy too into that functions
It’s happening for me too, unfortunately. Seems to be a bi-monthly occurance at this point. Really unfortunate, when it is affecting a production environment.
I’m also experiencing this issue today since around 4 or 5 hours ago with a function that runs every 5 minutes.
Is Azure violating the SLA with these interruptions in service? I have no clue what the difference between their app service plan and consumption plan SLA is but the wording is definitely different:
Same issue here, seems to affect only West Europe but it is very scary that our apps would just randomly become unavailable and we can’t do anything about it
Experiencing the same. Suddenly all envionment stopped working
encountering this right now - out of nowhere stopped working
Any news on this?! We are experiencing the same problems in West Europe region
Mine are running in West Europe as well, same for @Arjan321
Hi Arjan32,
Normally this error message is related with misconfiguration.
Please take a look the bullets below:
Add the variable
AzureWebJobsStorageand add the value of the connection string from the Storage Account Reference:Double check the value of the variable
WEBSITE_RUN_FROM_PACKAGEReference: