kudu: Zip deploy fails randomly on node_modules files for Linux web app

Have you tried WEBSITE_RUN_FROM_PACKAGE=1 env var?

It worked wonders for us (didn’t fix the symlink issue, but we circumvented that by specifying the full path.)

https://docs.microsoft.com/en-us/azure/azure-functions/run-functions-from-deployment-package#benefits-of-running-from-a-package-file

it says functions, but works with node on linux app service.

Azure Sucks… thats the problem. On AWS people can get work done plain and simple. Microsoft’s warped architectures does nothing but hold you back. Maybe we just need a few more layers of crap on top of the other layers of crap to fix it?

I’m pretty sure it’s connected with symlinks that are used in npm modules

@gecode-es Glad to hear it helped you!

Here is my understanding of azure web apps and kudu (I’m an azure customer, not an MSFT employee, so please correct me if any of these assumptions are wrong):

App Service runs kind of a docker container, however kudu sync treats the container as a mutable file system. That’s why at every deployment, you’re basically copying some of the files to the target file system. Also, you will see that if you create a file in the container, let’s say via touch foobar, that file will be preserved across deploys. So it does not start with a fresh container as in docker each time, it is more similar to how a normal machine would work, even though it is docker based.

With this env var though, nothing gets copied except the zip file itself, which is mounted as a filesystem, therefore it is atomic (all or none)

With kudusync, when you deploy a node app, it looks at the timestamps of each file in your app and node_modules, determines which files are newer, and copies those. This is problematic in node ecosystem as some packages have been built such that their timestamp is a fixed point back in time (1985 specifically.) node guys do this for reproducible builds, but this confuses kudu sync as it thinks that those files are either too old or the same.

Therefore, in general, mutating the file system is very problematic and error proen and this flag makes it as if a layer (somewhat like a docker layer) consisting of the contents of the zip have been applied.

So, it is a very clean approach and works almost without issues.

One of the issues we ran into was the following though, when the very same container is accessed via kudu’s rest api, kudu sees the system as if that zip file has not been mounted. We were using kudu rest api to run some one of commands, but when kudu access the app service, the folder is blank since it is not mounted.

Another issue was that the symlinks inside node_modules/.bin were not proper, so to run a migration via sequelize for example, we had to do the following:

node node_modules/sequelize-cli/lib/sequelize db:migrate

normally, npm run dbmigrate which is specified as sequelize db:migrate would work, but doesn’t and fails with the following issue: Error: Cannot find module './core/yargs'

In general, I recommend WEBSITE_RUN_FROM_PACKAGE=1 overall. One other benefit was that our deploys, which were taking 15-20 minutes due to node_modules now take 30-40 seconds.

For people still running into this while trying to deploy a Node app from Github, I thought I’d add the easiest work around I’ve found.

I just created a startup.sh script in the root of my projects which runs npm rebuild to recreate all the node_modules symlinks followed by my actual project’s run command. I then go into the app service configuration in the Azure portal and change the custom startup command to startup.sh.

This way the container will regenerate all of the symlinks before actually running your project. An example startup.sh script follows:

npm rebuild

npm run production

Hey guys,

we’re running Node JS application on the Linux machine (Azure Web App)

we’re using Travis CI to deploy to Azure Web Apps: https://docs.travis-ci.com/user/deployment/azure-web-apps/

It started to happen all of a sudden (the deployments worked up until 2nd October without problems)

Here are the logs:

2019-10-04T12:17:31  Updating branch 'master'.
2019-10-04T12:17:32  Updating submodules.
2019-10-04T12:17:32  Preparing deployment for commit id '44a1913ace'.
2019-10-04T12:17:32  Oryx-Build: Running kudu sync...
2019-10-04T12:17:32    Command: kudusync -v 50 -f /home/site/repository -t /home/site/wwwroot -n /home/site/deployments/44a1913ace81612c66a02c93e583a9e1985d770a/manifest -p /home/site/deployments/02061404a9d2c7007a1b55db5a60210542a4e217/manifest -i ".git;.hg;.deployment;.deploy.sh"
2019-10-04T12:17:34    Kudu sync from: '/home/site/repository' to: '/home/site/wwwroot'
2019-10-04T12:17:34    Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules'
2019-10-04T12:17:34    
/opt/Kudu/KuduConsole/Scripts/starter.sh kudusync -v 50 -f /home/site/repository -t /home/site/wwwroot -n /home/site/deployments/44a1913ace81612c66a02c93e583a9e1985d770a/manifest -p /home/site/deployments/02061404a9d2c7007a1b55db5a60210542a4e217/manifest -i ".git;.hg;.deployment;.deploy.sh"
2019-10-04T12:17:34  
2019-10-04T12:17:35  App container will begin restart within 10 seconds.

The unsatisfiable solution is to SSH into the machine, cd into the /home/site/wwwroot/ and remove whole node_modules folder (which is by the way present even though the Kudu sync command is complaining that the folder is missing) and rerun the Travis build (which is not how CI/CD should work)

Can someone explain me what could happen on 2nd October or why it stopped working without any change to deployment configuration? (it doesn’t work till now - tried to create a new service - first deployment passed and the second immediately failed)

We found a few issues with ZipDeploy :

1. Symlinks problem: The files in node_modules/.bin are symlinks. Depending on how users produce the .zip file for their deployment, it is likely that those symlinks are not included in the zipfile. npm automatically adds the files in node_modules/.bin to the PATH so they can be used, so if they are not in the .zip file the scripts might stop working. We have fixed this issue on Kudu for Linux, this should be available in our next platform release. Kudu would now recognize symlinks and defer their creation until the complete zip file is extracted. KuduSync ran into issues when Kudu failed to create these symlink files. If you have sample apps where you run into this issue frequently, let us know we can add this to our tests.

2. EACCESS issue : some of the users experienced this as well, files did not have correct permissions. Zip deploy now ensures that all the extracted files have correct file permissions.

3. One other possible cause is that the app is running and locking the binaries : We are considering versioned deployments to fix this - We recommend using slots for active sites currently.

We recently changed the fflow of Kudu based Node deployments and we compress the node_modules directory and extract it locally during runtime/mount it in the memory as a read only volume. (We use Microsoft Oryx to build artifacts on Kudu for Linux) . You could enable Kudu to build artifacts by enabling the app setting SCM_DO_BUILD_DURING_DEPLOYMENT= true. Or you could use RUN_FROM_PACKAGE feature, zip gets mounted on wwwroot as a read-only file system.

@pat-flew we don’t extract the zip package and recreate the symlinks. We use fuse-zip file system to mount this zip, when the app is run using RunFromPackage

I am having a similar issue when using WEBSITE_RUN_FROM_PACKAGE=1. Some of the symbolic links are not being set correctly during deployment

e.g.

/node_modules/html-webpack-plugin/node_modules/loader-utils/node_modules/.bin/json5

is being linked to

../../../json5/lib/cli.js

where it should be

../../../../../json5/lib/cli.js

consequently, the deploy fails with Error: ENOENT: no such file or directory, stat '/home/site/wwwroot/node_modules/html-webpack-plugin/node_modules/loader-utils/node_modules/.bin/json5'

@ustun some great insight thanks for taking the time to explain your understanding and experience. Really appreciated.

We are investigating the python Zip Deploy issue, the bug with node_modules has been fixed and patched across all the regions

@sanchitmehta Unfortunately, this hasn’t been fixed for node_modules. Just tried it again. Note the timestamp from the logs.

2019-10-22T16:46:42.9510471Z Error: EACCES: permission denied, mkdir ‘/home/site/wwwroot/node_modules/.bin’ 2019-10-22T16:46:42.9510611Z \n/opt/Kudu/Scripts/starter.sh kudusync -v 50 -f /tmp/zipdeploy/extracted -t /home/site/wwwroot -n /home/site/deployments/ca89f0184d7a41d5bd84e8218c8bfe4b/manifest -p /home/site/deployments/1ca21aa686af4aac9cfa661a145b4e77/manifest -i “.git;.hg;.deployment;.deploy.sh”

@nshumoogum We are looking at your app. @superfliege We have fixed this for the next Kudu Version. This fix would be a part of our next milestone which would start deploying in a week.

If you’re using Zip Deploy, a work around is to use Run From Package : Enable it by setting the app setting WEBSITE_RUN_FROM_PACKAGE=1

Same issue here for past week.

Error: EACCES: permission denied, mkdir ‘/home/site/wwwroot/node_modules/.bin’

Been using Azure for both .NET Core and Node projects for a few years now and I’ve never had any confidence in the services. I always deploy and worry that tomorrow something will break and I’ll find myself in a world of hell again. This latest issue is the final straw I think. 😦 my life is complicated enough without this nonsense.

@sanchitmehta Any update on the investigation of this? I encountered this issue last week and can’t seem to find a resolution. Switching to the Windows App Service isn’t an option for my current project, unfortunately.

I’ve had so much trouble with Azure App Service on Linux. It’s not a great experience setting up a node app there compared to deploying a .net application on a Windows service. I’m kind of upset about how much time I’ve wasted on this (I usually wouldn’t express this on a GitHub issue, but it’s a paid product). I’m concerned about what issues I may encounter in the future if setup was this bad.

For this specific problem I encountered, I managed to workaround this after a lot of trial and error.

Problem 1: After doing a deploy with symlinks, any deploy I tried after even without symlinks would fail, and web ssh didn’t work. Was getting a EHOSTUNREACH error. Solution: Open up the App Service in the portal, Development Tools -> Advanced Tools, then click on the “Bash” tab. For me, that worked for some reason and I was displayed a prompt. From there I was able to rm -rf the contents of the /home/site/wwwroot folder.

Problem 2: Since a deploy with symlinks wrecked the server, what could be done to not deploy a zip with symlinks? Solution: I tried a lot of different things (including different deployment strategies) and ran into so many different problems. The only thing that ended up working for me was to delete all the symlinks and copy the target of the symlink to where the symlink used to be, then zip the folder for deployment to the server. This wasn’t a big issue for me to do since the size of the files was small.

I used this powershell script in my azure pipeline to do this:

# Azure App Service Zip Deploy doesn't handle symlinks so inline
# all of them (https://github.com/projectkudu/kudu/issues/2946)
$links = dir . -recurse -force | ?{$_.LinkType}
foreach ($link in $links) {
  # store these before removing the link, otherwise the properties will go to null
  $source = $link.Target
  $dest = $link.FullName

  echo "Removing symbolic link at ${dest}..."
  (Get-Item $dest).Delete()

  echo "Copying ${source} to ${dest}..."
  Copy-Item -Path $source -Destination $dest -Recurse
}

@ustun - i am embarrassed, yes this worked for me; i should have double checked the variable name

Thanks everyone who tried to help me 😃

@nshumoogum WEBSITES_RUN_FROM_PACKAGE should be WEBSITE_RUN_FROM_PACKAGE

node_modules is locked and can’t be deleted via SSH or Bash on Linux machine 😦 swapping slot with an empty one deletes everything, here is my workaround for Node app zip deploy

az webapp deployment slot create --name YOURAPP --resource-group YOURGROUP --slot temp

az webapp deployment slot swap -g YOURGROUP -n YOURAPP --slot temp --target-slot Development

az webapp deployment slot delete --name YOURAPP --resource-group YOURGROUP --slot temp

works like a charm

@sanchitmehta Had the same as error @nshumoogum with a python app failing because of symlinks. Adding the ‘–copies’ option to venv creation fixed it! Hopefully adding WEBSITE_RUN_FROM_PACKAGE will add some consistency as well with other issues for us. Thank you and thanks for the transparency on these issues.

@nshumoogum If you’re using Python 3 and creating the environment yourself you could try adding –copies : python -m venv antenv --copies.

From your logs it doesn’t look like Run From package is enabled and Kudu is trying to do a normal zip deployment and extracting the node_modules. Could you please check the Kudu Environment page, if the app setting WEBSITE_RUN_FROM_PACKAGE=1 is present and then redeploy.

@willmtemple this could also be that a different user (kudu runs as a dedicated user/group) that is creating installing npm modules in the pipeline and kudu user does not have access to override them. You could add npm config set unsafe-perm=true to allow all users to override these files.

One question about node_modules (maybe this isn’t the right place to ask) – I don’t commit my node_modules when I deploy, and then my startup script performs an “npm install” before starting up the node process. Is this the correct approach?

Ideally you should have a build step when you zip everything and use that during release (deploy) stage, otherwise it will be flaky. Also you should not use npm install but npm ci since npm install does not honor package-lock.json.

Same issue. Deploying node app to Web App service on Linux stopped working recently. Zip deploy fails
Error: EACCES: permission denied, mkdir ‘/home/site/wwwroot/node_modules/.bin’

is this Azure issue or kudu? Any ETA on fix?

No real issues with .NET Core till now but this past week makes me wonder how much Microsoft ❤️ Linux at all. Very frustrating.

👍 same issue here with nodejs express app deployed by github repository

facing same issue via Local Git deploy - EAST US + EAST US 2 regions. @sanchitmehta Is there an update for my region??

@sanchitmehta Thanks for the update

For others, we’ve been able to work around this until the fix is in place by deleting out the /home/site/wwwroot/node_modules folder by running rm -rf node_modules from the wwwroot dir

It seems I have exactly the ZipDeploy Symlinks problem. Currently I can’t publish my node app on Linux. RUN_FROM_PACKAGE is not available on Linux. FTP upload would take over a day. When will this be fixed?

I was having a similar problem with a PHP project that is using composer:

Error: ENOENT: no such file or directory, stat ‘/home/site/wwwroot/vendor/bin/yii’

I had to remove the folders with symlinks from the zip package:

- task: DeleteFiles@1
  displayName: Delete vendor/bin folder from the staging directory
  inputs:
    sourceFolder: $(Build.StagingDirectory)
    contents: vendor/bin

I am having this exact same issue, and none of the suggestions are working for me. I suspect it has to do with Symlinks as well, but I can’t confirm that.