azure-functions-host: Unable to perform slot swap for Premium Function with network restricted storage account
Cannot perform a slot swap operation on a Function App in an Elastic Premium App Service Plan when the underlying storage account has network restrictions. When a slot swap is performed a 500 Internal Server Error is returned with the error message “There was an unexpected error swapping slots ‘staging’ and ‘production’ for site ‘<sitename>’. Please try to cancel your swap operation. ExtendedCode: 04093”. In the Diagnose and solve problems blade there is a further error: “Swap failed. Details: System.ServiceModel.Web.WebFaultException`1[Microsoft.Web.Hosting.Administration.ErrorEntity]: Bad Request (Fault Detail is equal to Code: BadRequest, ExtendedCode: Storage access failed. Storage volume is currently in R/O mode, Message: 04234).”
The storage account has firewall rules enabled restricting access to certain IPs and the function app connects via a private endpoint using vnet integration.
When I disable the network restrictions the slot swap operation succeeds.
Check for a solution in the Azure portal
No solution in the Azure Portal.
Investigative information
Please provide the following:
- Timestamp: 2022-06-08T03:12:00Z
- Function App version: 4
- Function App name:
- Function name(s) (as appropriate):
- Request ID of swap operation (unsure if this is the same as Invocation ID): 74fe4110-28cd-4fd0-8262-d5b1b39a1ad8
- Invocation ID:
- Region: Australia East
Repro steps
Provide the steps required to reproduce the problem:
- Create function app with deployment slot inside Elastic Premium App Service Plan with network restricted storage account (setup is similar to this sample template but with a deployment slot)
- Perform slot swap
Expected behavior
Slot swap should succeed.
Actual behavior
Slot swap fails with error “There was an unexpected error swapping slots ‘staging’ and ‘production’ for site ‘<sitename>’. Please try to cancel your swap operation. ExtendedCode: 04093”. In the Diagnose and solve problems blade there is a further error: “Swap failed. Details: System.ServiceModel.Web.WebFaultException`1[Microsoft.Web.Hosting.Administration.ErrorEntity]: Bad Request (Fault Detail is equal to Code: BadRequest, ExtendedCode: Storage access failed. Storage volume is currently in R/O mode, Message: 04234).”
Known workarounds
Slot swap succeeds when network restrictions are removed from the storage account.
Related information
Provide any related information
- Programming language used: C# .Net 6
- Links to source
- Bindings used
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 21 (9 by maintainers)
Hi everyone, apologies for the delay in updating this issue, we went through an extra load of testing before sharing. The fix is deployed but we had to introduce a new app setting that you should set on your production slot (or the swap slot if you’re swapping between two subslots) called WEBSITE_OVERRIDE_STICKY_DIAGNOSTICS_SETTINGS and set it to 0 (zero). I.e.,
This will allow you to swap the slots when the storage account is network restricted. Here is our documentation on app settings. This should not have any impact on your Azure Monitor related diagnostics settings configuration and is related to the legacy Application Log Settings configuration, which was preventing Premium Functions slot swaps from occurring.
Next steps on our side are:
This was caused by an internal platform component, and I’ll keep the issue open to notify when the component fix has been fully released. Unfortunately, the ETA for a full roll out is within the next 3 to 4 months.
Looks like this is a known issue with a bug fix completed, it will be in the next release but there isn’t an ETA on that right now. This is an an underlying platform issue and not a Functions specific issue so there isn’t anything we can do here but wait for the release. I can keep this open to track that the bug fix addresses the issue and let you know when the platform release is out.
The only available workaround would be to disable the firewall while swapping.
@liliankasem Hi Lilian, please kindly go through chat content on Teams. This Github issue is related to a support ticket and please do let me know if anything is needed. Cheers.
Hi everyone, we are in final testing phase to ensure we have addressed completely and will work on documenting the steps so you can complete the swap without issues when storage is network restricted. Hoping to have an update in the next couple of weeks.
@nzthiago I’ve attempted any number of firewall and network configurations to see if I could successfully complete a swap. However, all of them have reported the same response below when investigating the issue in the support blade.
I’ve opened support ticket 2210060010000492 in response as requested.
Same issue here. A more specific ETA would be greatly appreciated.
The ETA remains as per above, between September and October.
Perfect thanks that for that @liliankasem