azure-functions-dotnet-worker: upgrading Microsoft.Azure.Functions.Worker.Sdk from 1.15.1 to 1.17.0 causes build to fail

Description

MSB4018: System.IO.DirectoryNotFoundException: Could not find a part of the path ‘C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\obj\Release\net48\WorkerExtensions\buildout\bin\extensions.json’.

this problem is similar to https://github.com/Azure/azure-functions-dotnet-worker/issues/2125.

i use teamcity to build my solution with the following command:

"C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe" C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.sln -t:Rebuild -p:Configuration=Release -p:TargetProfile=Production

my azure function is a v4 dotnet-isolated project targeting net48.

i am running the latest tools (as of this post):

PS C:\> func
Azure Functions Core Tools
Core Tools Version:       4.0.5530 Commit hash: N/A +c8883e7f3c06e2b424fbac033806c19d8d91418c (64-bit)
Function Runtime Version: 4.28.5.21962

when my azure function project includes:

<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.15.1" />

the build succeeds.

after upgrading to:

<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.0" />

the build fails.

here are the relevant lines from the build log of the successful build:

all references to extensions.json:

[19:21:27] :	[Project "WorkerExtensions.csproj": Build]   Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.Console.dll "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin" "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\extensions.json"
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\extensions.json" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\extensions.json".
[19:22:40] :	[Project "HoursTracker.Jobs.Host.csproj": Rebuild]  [34;1m  Deleting file "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\extensions.json".
[19:22:45] :	[Project "WorkerExtensions.csproj": Build]   Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.Console.dll "C:\TeamCity\buildAgent\temp\buildTmp\cane5v32.jmv\buildout\bin" "C:\TeamCity\buildAgent\temp\buildTmp\cane5v32.jmv\buildout\bin\extensions.json"
[19:22:46] :	[Project "HoursTracker.Jobs.Host.csproj": Rebuild]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\cane5v32.jmv\buildout\bin\extensions.json" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\extensions.json".

and the entirety of WorkerExtensionsBuildCopy task:

[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [36;1m_WorkerExtensionsBuildCopy:
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Creating directory "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Creating directory "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Creating directory "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Creating directory "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Azure.Identity.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Azure.Identity.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Azure.Core.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Azure.Core.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Azure.Storage.Queues.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Azure.Storage.Queues.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Azure.Storage.Common.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Azure.Storage.Common.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\extensions.json" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\extensions.json".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\function.deps.json" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\function.deps.json".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Azure.Functions.Worker.Extensions.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Azure.Functions.Worker.Extensions.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Azure.Functions.Worker.Extensions.pdb" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Azure.Functions.Worker.Extensions.pdb".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Azure.WebJobs.Extensions.FunctionMetadataLoader.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Azure.WebJobs.Host.Storage.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Azure.WebJobs.Host.Storage.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Bcl.AsyncInterfaces.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Bcl.AsyncInterfaces.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Azure.WebJobs.Extensions.Storage.Queues.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Azure.WebJobs.Extensions.Storage.Queues.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Extensions.Azure.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Extensions.Azure.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Identity.Client.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Identity.Client.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.IdentityModel.Abstractions.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.IdentityModel.Abstractions.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.Identity.Client.Extensions.Msal.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.Identity.Client.Extensions.Msal.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\System.IO.Hashing.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\System.IO.Hashing.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\bin\Microsoft.WindowsAzure.Storage.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\Microsoft.WindowsAzure.Storage.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Creating directory "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\runtimes\win\lib\netstandard2.0".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\temp\buildTmp\fnfxvons.qa2\buildout\runtimes\win\lib\netstandard2.0\System.Security.Cryptography.ProtectedData.dll" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\.azurefunctions\runtimes\win\lib\netstandard2.0\System.Security.Cryptography.ProtectedData.dll".
[19:21:28] :	[Project "HoursTracker.Jobs.Host.csproj"]  [36;1mDone Building Project "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\HoursTracker.Jobs.Host.csproj" (default targets).

here are the relevant lines from the build log of the failed build:

all references to extensions.json:

[19:10:49] :	[Project "WorkerExtensions.csproj": Build]   Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator.Console.dll "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\obj\Release\net48\WorkerExtensions\buildout\bin" "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\obj\Release\net48\WorkerExtensions\buildout\bin\extensions.json"
[19:10:50] :	[Project "HoursTracker.Jobs.Host.csproj"]  [34;1m  Copying file from "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\obj\Release\net48\extensions.json" to "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\extensions.json".
[19:11:58] :	[Project "HoursTracker.Jobs.Host.csproj": Rebuild]  [34;1m  Deleting file "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\bin\Release\net48\extensions.json".
[19:11:58] :	[Project "HoursTracker.Jobs.Host.csproj": Rebuild]  [34;1m  Deleting file "C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\obj\Release\net48\WorkerExtensions\buildout\bin\extensions.json".
[19:11:58] :	[Project "HoursTracker.Jobs.Host.csproj": Rebuild]  [34;1m  Deleting file "obj\Release\net48\extensions.json".

and the build error:

[19:12:06]E:	[MSBuild] C:\TeamCity\buildAgent\temp\buildTmp\.nuget\packages\microsoft.azure.functions.worker.sdk\1.17.0\build\Microsoft.Azure.Functions.Worker.Sdk.targets(166,5): error MSB4018: System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\TeamCity\buildAgent\work\d3709b43d4e86d98\Source\HoursTracker.Jobs.Host\obj\Release\net48\WorkerExtensions\buildout\bin\extensions.json'.

Steps to reproduce

upgrade Microsoft.Azure.Functions.Worker.Sdk from 1.15.1 to 1.17.0 and compile the project using msbuild.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 36 (14 by maintainers)

Most upvoted comments

I am facing the same issue when building function app project targeting .NET 8 in isolated model in Azure DevOps pipeline. The project build fine in local environment but gives this error in Azure DevOps pipeline. I have added steps for installing both .NET 6 and .NET 8 in the ADO pipeline but still facing the same issue. What is the solution to this problem?

Here is the error message : ##[error]C:\Users\VssAdministrator.nuget\packages\microsoft.azure.functions.worker.sdk\1.17.2\build\Microsoft.Azure.Functions.Worker.Sdk.targets(190,9): Error MSB4018: The “EnhanceExtensionsMetadata” task failed unexpectedly. System.IO.DirectoryNotFoundException: Could not find a part of the path ‘D:\a\1\b\publish_output.azurefunctions\extensions.json’.

@kshyju .NET 6 should not (and can not be required) for some automatic generated interal project if we are building an Azure functions project that is based on .NET 8

@fijsh do you get the same error in 1.17.3-preview1? If so, can you share your build steps? Are you setting OutDir as well?

I was able to fix this by deleting all Nuget caches on my machine. I use Rider so there you go to Nuget->Folders tab and clear all caches there. 1.17.2 as well as 1.17.3-preview don’t give an issue anymore. I was having this on a local build.

Build steps in the pipeline are:

          - task: DotNetCoreCLI@2
            displayName: dotnet restore
            inputs:
              command: restore
              projects: '**/*.csproj'
              feedsToUse: 'select'
              vstsFeed: 'FeedName'
              includeNuGetOrg: true
          - task: DotNetCoreCLI@2
            displayName: "Build Projects"
            inputs:
              command: 'publish'
              projects: '**/*.csproj'
              arguments: '--configuration Release --output publish_output'
              publishWebProjects: false
              modifyOutputPath: false
              zipAfterPublish: false

@jaywar: 1.17.2 had many of the SDK changes rolled back due to a regression (which is addressed, and the SDK changes added back in 1.17.3-preview1). The major difference between the two versions, and what leads to your issue, is the inner build directories have been overhauled. Before we always got a new temp folder to build in, so your projects build dir never influenced it. The new way uses your intermediate output path (typically the obj folder).

I think this must have something to do with a minute difference between Dotnet and VSBuild tasks. One thing I can tell is the output property does differ between them: --output will set OutputPath in the build, and VSBuild you are setting OutDir. I need to look into if OutDir is a property we should be expecting/supporting.

I do recommend your approach of using dotnet CLI over nuget & vsbuild tasks. DotNetCoreCLI tasks are the recommended way to build C# projects in ADO

An interesting side effect of this affects us in our samples at Particular Software at https://docs.particular.net.

We multi-target our samples for ease of development, and when we build our docs site a process munges the csproj files to create individual zip file downloads for each target framework.

Now, I suppose that when the outer (multi-targeted) build triggers the inner (single-target) build it does so with $(TargetFramework) value that bleeds through to the additional project (what @kshyju refers to as the “inner” build that shows up in the build directory as the .azurefunctions directory - I prefer to call it the “host” project) and prevents our build from succeeding.

So while we know it’s not a “normal” use case for a Functions app to be multi-targeted (it’s an app, you have to pick something to run it on!) up until this point it helped us to be able to do it. But I do wonder if there isn’t a way to change what’s going on in MSBuild that would allow the “host” project to dictate it must build as net6.0 without putting any extra requirements on the environment, which would make others in this thread happy and allow our multi-targeted builds to succeed as well.

@kshyju .NET 6 should not (and can not be required) for some automatic generated interal project if we are building an Azure functions project that is based on .NET 8

.NET 6 is required because the functions host runs on .NET6 - note that the host is a distinct process from your worker (what your csproj builds) which can be .NET8. Our SDK has an inner build which is responsible for discovering and collecting the host extensions, which must be all done with the .NET6 TFM. If any other TFM is used for this inner build you risk encountering runtime assembly load failures. While this is not ideal, there is no way around it. The host is a prebuilt application and the TFM used by it is intentionally decoupled from what TFM you use in your worker project. This is so you do not need to wait for the host to support a TFM before you yourself upgrade to it. The result of this is that we still need the TFM for the host available at both compile and runtime.

@kshyju i appreciate your follow up. i have subscribed to and read through all of the threads reporting similar issues and seen your copy/pasted response in all of them. i’ve reviewed the “regression” and i’ve looked at the reversion of the commit. i understand the differences in the different versions.

as for my project, i do have a TargetFramework variable set. it’s always been there. always worked. in 1.17.2 it doesn’t. frankly, i’m not sure that i can remove that variable. time permitting, i will explore why i have it there. i’ve had this build set up for years.

in the meantime, i’m sitting on an older version until you guys get this sorted out. the internet has clearly spoken and is not happy with the hardcoded net60 dependency on the internal project.

@davidpeden3 The 1.17.0 & 1.17.1 versions had a regression. 1.17.2 fixed that (rolled back the changes that caused the regression).

When you build a dotnet-isolated function, it creates an inner project called WorkerExtensions.csproj in a temp folder (You can see the full path of it in the build output). The TargetFramework of this project must be net6.0. Based on the error you are seeing , looks like this value was overridden by some other settings.

Do you have a global.json in your repo? Can you think of any other thing in your CI which is forcing the TargetFramework value to be different than net6.0?

Do you see same error when trying to publish it locally by running dotnet publish -c release ?

If you want to try with a minimal .NET framework app, you can download this app and try that. I was able to create a GH action to publish this minimal project without any issues. You can see the GH action script here.