kudu: Receiving "Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)" when deploying our app

Update

We are currently investigating. It is likely due to recent update on the VMs leading to this. It affects “West Europe”, “West US 2”, “North Central US” and “West Central US” regions.

You can mitigate yourself by performing either of the below.

1. Restarting Kudu process. Browse to SCM site -> Process Explorer and Kill w3wp (SCM). The new w3wp will start up automatically. If you have multiple instances, you may need to use instance dropdown to iterate and do the same on all instances.

2. Goto portal. Navigate to site blade, click Restart. This unfortunately will cause short down time for the site.

3. Scale up and scale down. This indirectly restarts SCM sites.

We are also gradually mitigate and will provide ETA once we know it. Apology for the inconvenience.

For everyone having this problem, please set SCM_DO_BUILD_DURING_DEPLOYMENT to false in your app service settings and thank me later

I had this issue and all I did was go into the Deployment Center, click on “Disconnect” and then re-deploy and it worked

Had the same issue today. The service plan had plenty of room left (CPU/Memory/Disk), no other deployments were being processed at the time and a run that previously succeeded failed when retried today (same commit). Additionally it seemed that kudu crashed as well or something, the suggested link for more debug information gave a 500 and other attempts to access kudu, logs or files of the function all failed (the ones I tried at least).

In the end I managed to resolve the issue by deleting the function app, and re-creating it (same settings). Thankfully this was a (new) function in a development environment, this would be an unacceptable workaround on a live production environment.

I have solved the problem by scaling up the service plan where my azure functions were. I have switched from B2 to B3 tier, which only increase RAM and ACUs so I’m guessing that we had consumed all ACUs ?

Since today all our applications cannot be deployed, some applications receive a 500 while others receive a 409 response. Like @i3anaan pointed out deleting the function app and re-creating it resolves the issue. We cannot do this in prod since it will result in tens of minutes of down time

same issue since today here (same scenario than @i3anaan ), any solution other than recreating everything?

First off, this should not impact Linux App Service - if you encounter this on Linux, please open a service ticket or http://aka.ms/CRI-WebApps. As for Windows, you should only experience this during deployment and, once manual mitigation by restart/scale, you should not encounter it again. As for the rollout, we have fix about 50% on the envs - it may take another week or so to rollout to the rest. Apology for the delay.

This has cropped up for us again. It seems that large deployments are just not really reliable with Azure Web Apps. Deployment logs don’t really help either. We have thousands of files in the deployment, but usually only one or two files have changed when we deploy, and it looks like Kudu just doesn’t cope well with that scenario.

ok. Found the issue - it is related to our app service plan storage limit - we were right at the limit of 10GB due to some logs. This would explain why changing the app service plan seems to be a very good workaround to this issue. Would be great if KUDU shows a proper error for this, seems that there are many people spending many hours investigating these issues.

Still happenning to us on linux app service

Ran into the same issue today in west europe!

Error: Failed to update deployment history. Conflict (CODE: 409)

• Location: West Europe
• App Service Plan: P2v2: 1
• Operating System: Linux
• Build: GitHub Action

[group]Run azure/webapps-deploy@v2
with:
  app-name: some-app-name
  slot-name: dev
  publish-profile: ***
  package: .
env:
  pythonLocation: /opt/hostedtoolcache/Python/3.8.7/x64
[endgroup]
Package deployment using ZIP Deploy initiated.
[error]Failed to deploy web package to App Service.
[error]Deployment Failed with Error: Error: Failed to deploy web package to App Service.
Internal Server Error (CODE: 500)
[warning]Error: Failed to update deployment history.
Conflict (CODE: 409)

Lets see if they can do something… https://twitter.com/jotajota930621/status/1288503705994764290

Experiencing the same problem here in a DevOps Release pipeline of App Service and Windows Function App projects. We first saw this error occur on 21/07/20 but seeing several occurrences today. The region for both the App Service and Functions is West Europe.

2020-07-29T09:50:50.1288995Z Trying to update App Service Application settings. Data: {"WEBSITE_RUN_FROM_PACKAGE":"1"}
2020-07-29T09:50:50.4010044Z App Service Application settings are already present.
2020-07-29T09:51:46.6646202Z Package deployment using ZIP Deploy initiated.
2020-07-29T09:52:53.1693953Z ##[error]Failed to deploy web package to App Service.
2020-07-29T09:52:53.1707005Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

A possible workaround seems to be to change the deployment method to Zip deploy, but for one of our Application Services this has increased the deployment time from 1 minute to 21 minutes which is not going to be acceptable.

Totally understand the pain and appreciate the patience. We will continue to roll out fix to address this - ETA end of Aug 2020.

This also started happening for us today on S1 tier in multiple App Service Plans. Scaling up to S2 tier solved it. After scaling back down to S1, I was still able to release.

Out typical ASP usage on S1 is 6% CPU and 60% memory.

Interesting note, which may not be relevant: in one ASP, I changed the tier up and then down again without doing a release in between. This did not solve the issue. The workaround has only fixed the issue if I do at least one release to the ASP at the new tier, before reverting it.

Edit: Should say thanks to @jjavierdguezas for the idea of changing tier

My deployments are passing, but I am continuously seeing the Error: Failed to update deployment history. Error: Conflict (CODE: 409) warning. In the line before that in my logs, I see "There is an auto swap deployment currently ongoing please try again when it completes." I can’t figure out what deployment is ongoing. I am using auto swap and so my pipeline does not run any commands specific to the deployment slot, besides actually deploying to it. I have tried playing around with adding the “Complete Swap” task to my deploy job, but it doesn’t seem to make a difference. Anyone else encounter this issue with a successful deployment but a warning about failing to update deployment history?

I’m having it since yesterday on Linux based functions in west europe, no consumption, B2/ 1-3 instances makes no difference

Having this issue frequently. Seems random when it works or not. Tried increasing to P3V2 sku and still no luck. Our development App Service works every time but the production one fails 2-3 deploys. We are on Canada Central.

I’m having this issue right now. Linux App Service @ East US.

We just ran into this issue on a production service and managed to resolve it in the short term. There are few things going on. First, there is a storage limit across all app services in a region and resource group. In our case we hit 500 GiB across all the app service plans. Our plans were all P3v2, so we had no ability to scale up/down when this happened, as we were at the highest tier.

In addition, we were trying to run multiple DevOps pipelines to multiple applications at the same time. When a deployment happens some storage is allocated to allow for the deployment to complete. In our case this allocation was enough to put us over the limit. We found if we continually monitored the storage quota across the region and resource group it eventually freed up enough storage where we were able to deploy.

Our long term fix is to identify the applications that are using a large amount of storage and determine how we can either limit the amount of storage each one of them is using, or provision some of those applications into new resource groups so the limit isn’t a concern.

Still happenning to us on linux app service

We’re also experiencing this issue and are currently manually invoking the scale-up/scale-down workaround for a couple of our client’s production apps.

@suwatch - Was the fix for this rolled out on time?

Same issue here

Linux machine running an ASP.NET Core 2.2 web app at West Europe

The Release Azure DevOps process to deploy the App on Azure was running successfully, but suddenly we have received an error during the automatic deploy process that is being executed on Azure DevOps:

##[error]Failed to deploy web package to App Service. ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409) ##[warning]Error: Failed to update deployment history. Error: Bad Request (CODE: 400)

My steps to fix the problem as workaround:

• Scale up the App Service Plan on Azure
• Execute from Azure DevOps the Release process manually
• Scale out the App Service Plan again.

This thread and comments are correct and saved my time and problems in a critical issue.

Thanks all.

@suwatch I sent you an email with the information to the address in your profile, but since I scaled up the app service we have not seen the error anymore

@asmendez Did the first deployment fail for some reason ? We have a cool off period in Deployment Locks (when an unexpected error occurs leading to a Kudu Crash) to avoid conflicts. Could you share your app name ?

@mcdurdin are you still facing this problem ?

Experiencing this issue as well. Bit of context: Python FA, so we have to use a Linux plan.

• Initially, we had a Linux consumption plan and ran from package with a URL (only way that is supported for Linux Consumption according to this).
• Due to reasons, we switched to a dedicated plan, S1
• We started getting this error, and diving in the logs, I came across Run-From-Zip is set to a remote URL using WEBSITE_RUN_FROM_PACKAGE or WEBSITE_USE_ZIP app setting. Deployment is not supported in this configuration. Seems weird, it doesn’t say that Linux with dedicated plan does not support URL, but ohwell
• We switched to WEBSITE_RUN_FROM_PACKAGE=1, but that causes the wwwroot folder to become readonly, which doesn’t work for us
• So in the end we switched to just zipDeploy in our AzureFunctionApp@2 CI/CD pipeline. It started taking quite a long time (~15 minutes, big zip, I think because the zip contains the complete python environment including all dependencies, so around 18k files), but in the end works
• … Sometimes. And sometimes we still get this error

Microsoft warns about zipDeploy being a bit less atomic than runFromPackage (1 2), so maybe that’s also where the 409 conflict comes from? The deploy might be OK and finished, but maybe under the hood there is still something happening that causes a fast next deployment to fail?

It’s worth noting for us the conflict was due to the service hook in Azure DevOps sticking around from when deployment was setup previously. We needed to manually delete it in Azure DevOps in order to get the conflict error to go away

Had this issue today as well in west europe. It deployed fine on our test subscription, but as i tried to deploy it in our production subscription it failed saying that the requested features were not supported. After reading all of this i tried changing the plan from S1 to S2 and then it deployed right away, and could update the plan back to S1 after.

@ransagy @petro2050 we found two possible solutions, which have solved it since a while: • do not deploy very fast • ensure that if you run on a service plan the config of the asp is good enough as the deployment seems to need more than just running the code

perhaps that helps 😊

I’m having this issue all of a sudden. Deploys were fine for forever (at least the last year), then all of a sudden starting to see this. A quick google search brought me to this thread. Repeated deployment attempts eventually succeed. I haven’t tried the scale up and down solution as a work around yet. If I find some motivation in the next few days I’ll see if I can gather some more details to throw in the pile here.

I wanted to circle back on this (after getting some notifications about these recent comments on the thread) My issues were resolved by fixing a completely unrelated issue that happened to manifest this way. It’s been…a while…since I got it fixed and can’t for the life of me remember the specifics, other than it had absolutely nothing to with Kudu despite the issue manifesting as a Kudu deploy failure. So, not the most useful update, but I don’t have problems anymore 😂

I’m having this issue all of a sudden. Deploys were fine for forever (at least the last year), then all of a sudden starting to see this. A quick google search brought me to this thread. Repeated deployment attempts eventually succeed. I haven’t tried the scale up and down solution as a work around yet. If I find some motivation in the next few days I’ll see if I can gather some more details to throw in the pile here.

I have the same problem. but i could solve it by stopping the app-service, starting it again and then restarting it 3 times in quick succession. This seems to stop the w3wp and the release is working again.

We received the same issue since Friday last week for one of our App Services in West Europe, using Zip-Deploy. We tried to do the following:

1. Restart the App Service and re-deploy our code via our Azure build pipeline => still received the 409 error
2. Stop the App Service re-deploy our code via our Azure build pipeline => still received the 409 error
3. Kill all w3wp processes in the SCM console and re-deploy our code via our Azure build pipeline => still received the 409 error
4. Delete the App Service, re-deploy the App Service via our ARM templates, stop the App Service, and re-deploy our code via our Azure build pipeline => success!

So only deleting the entire App Service and re-deploying fixed the issue.

Today we faced this issue in another subscription, this time I restarted the app (as @suwatch said) and then the release went just fine. So, no scaling is really needed… good news to save money… 😅

From our telemetry, we observed below exception happening to multiple sites and vms started a few days ago. We are unable to repro. If anyone is still stuck in this state, please try to go to portal, click restart the site (assuming you can afford site to be down for such short period). Try zipdeploy again - and see if that mitigate the issue.

Error occurred type="error" text="The type initializer for '<Module>' threw an exception." stackTrace="   at Newtonsoft.Json.Converters.BinaryConverter.CanConvert(Type objectType)
   at Newtonsoft.Json.JsonSerializer.GetMatchingConverter(IList`1 converters, Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.InitializeContract(JsonContract contract)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.CreateObjectContract(Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.CreateContract(Type objectType)
   at Newtonsoft.Json.Serialization.DefaultContractResolver.ResolveContract(Type type)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.Linq.JToken.FromObjectInternal(Object o, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.Linq.JObject.FromObject(Object o, JsonSerializer jsonSerializer)
   at Kudu.Core.Infrastructure.LockFile.WriteLockInfo(String operationName, Stream lockStream) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\LockFile.cs:line 213
   at Kudu.Core.Infrastructure.LockFile.Lock(String operationName) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\LockFile.cs:line 153" innerText="Attempted to read or write protected memory. This is often an indication that other memory is corrupt." innerStackTrace="   at <CrtImplementationDetails>.LanguageSupport.Initialize(LanguageSupport* )
   at .cctor()

@grantwar Are you still experiencing the same problem ? Could you provide us your app name indirectly ?

@sanchitmehta Yes I’m still experiencing the issue, the app is failing when I’m trying to deploy to both my P2V2 and S1 service plans.

I have another app name if you need it

I would be great if you all can share the webapp name - this would help us pinpoint.

I’m experiencing the same problem. 2020-07-29T15:06:45.5681580Z ##[error]Failed to deploy web package to App Service. 2020-07-29T15:06:45.5701015Z ##[error]Error: Error: Failed to deploy web package to App Service. Conflict (CODE: 409)

If you are on a free subscription plan, make sure you haven’t reached your storage limit which is 1gb. If so, you have to upgrade to a paid one…

same problem here