azure-functions-host: Could not load file or assembly Microsoft.IdentityModel.Tokens (related to _FunctionsSkipCleanOutput)

Commenting to keep issue open

Same issue here - .net6, functions runtime v4. Any version above 6.10.2 fails to load. Given the upcoming EOL for .Net Core driving migrations to .Net 6, and the ever present need to remain up to date with security fixes in any app, being unable to keep a security-focused library up to date without workarounds while migrating to .Net6 is very frustrating.

Also ran into this trying to reference the latest Microsoft.IdentityModel.JsonWebTokens library at version 6.15.0 It can be repro’d by creating a new azure functions project in vs code using the azure functions extension. Select dotnet 6.0 (in-process) and runtime host of v4. If I try and reference any objects from that library I get the dreaded Could not load file or assembly error. I did quite the bit of research. Most roads point to the saga that is #5894, and the _FunctionsSkipCleanOutput suggestion there DID work.

However - as others have said - it is a bit of a club. The whole all or nothing approach to cleaning the output. I took a look at the FunctionsPreservedDependencies feature that was added in pull 6849 in order to address this defeciency, and it didn’t work for me.

Lastly, I did notice that if I pin my Microsoft.IdentityModel.JsonWebTokens version to 6.10.0 everything worked just fine. 🎉

Full example Azure functions csproj below

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <AzureFunctionsVersion>v4</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.IdentityModel.JsonWebTokens" Version="6.10.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.0.1" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

Sharing this pin to 6.10.0 solution for other folks that don’t may not want to use _FunctionsSkipCleanOutput or FunctionsPreservedDependencies. My guess is this works because the azure functions host is currently using this version? Can see it listed in this repo here. Sure would be nice to be able to reference the latest 6.15.0 Microsoft.IdentityModel.JsonWebTokens and it’s security enhancements.

Burned a day researching this. Looks like @anthonychu has got feature request 6650 that might help folks save some research time on this?

Apologies all for the troubles caused by these automations. And my thanks to those commenting to keep the thread in the correct state, which is Open. I share in your frustration around this. I’ve removed that label so the bot shouldn’t keep trying to do this. I’ll follow up with the team about fixing this process in general.

Apologies all for the troubles caused by these automations. And my thanks to those commenting to keep the thread in the correct state, which is Open. I share in your frustration around this. I’ve removed that label so the bot shouldn’t keep trying to do this. I’ll follow up with the team about fixing this process in general.

Any update on a fix in progress or ETA @mattchenderson ?

What is going on with this issue and all the recent attempts to be closed without being solved? I have to say it, but as of lately I’ve been getting slightly disappointed at Microsoft, this is not the first nor the only issue I’ve witnessed in recent times.

encountering this issue also. Will also be doing as mentioned in 7601 and adding this to all future projects until this is fixed.

Hi Guys, just want to add, in my case I can update up to ‘Microsoft.IdentityModel.Tokens, Version=6.24.0.0’, Higher than that will produce the same ’ Could not load file or assembly ’ error.

It works if I use the workaround “_FunctionsSkipCleanOutput” as mentioned above.

@Bindsi you may follow the “workaround” in the original post,.

Known workarounds

Add <_FunctionsSkipCleanOutput>true</_FunctionsSkipCleanOutput> to the .csproj file. This is a workaround, not a solution, because this property is internal and undocumented.

It does work indeed, but we should remain vigilant for an actual fix so that we don’t need the SkipCleanOutput property set to true.

Addressing the point about security upgrades, I agree that a silent snap back would be bad in the abstract. That idea would need to be paired with us always being on latest from our side. To be clear, we do have a process for handling CVE-related updates. What’s missing is the handling of other classes of update. We could additionally automate any update to the given assembly, beyond what we already do with our CVE handling. The deployment process means there would be a small window of time where we would lag behind latest, again outside of CVE response.

Technically we could do auto-upgrades without going to full unification, but in those interim windows, an upgrade and publish from the app could still hit the runtime error. Unification would at least prevent that error from coming up for that interim period. It is a snap-back, but with the auto-upgrade process, it wouldn’t be pulling back far, and it would only do so for a short while.

Taking a step back, here’s how I’m thinking about the behaviors:

When I say “keep local assemblies” above, I am referring to the _FunctionsSkipCleanOutput and FunctionsPreservedDependencies options. Between the two, we would recommend FunctionsPreservedDependencies since you can specify indvidicual assemblies. Since I don’t think an example of that has been captured in this thread:

<ItemGroup>
<FunctionsPreservedDependencies Include="Microsoft.IdentityModel.Tokens.dll" />
</ItemGroup>

There is a gap in official documentation for these properties, and that’s something we need to address.

The ideal state, I think, is for this to make the host reference be a fully private assembly. Unfortunately, that would be a breaking change because apps may be relying on the host providing the version. This therefore cannot be introduced within V4. However, at any future point in which we would be making a breaking change to the host anyway, this would be the right change to make. A variation might be to move any behaviors around it to an extension rather than being directly in the host. In that case, there is technically no “host version” to speak of - it’s all the payload version because the user controls the extension.

But in the context of V4, our options to avoid the reported issue are to have folks continue using the workaround or to move to full unification. If we can establish an auto-upgrade process for new versions of the identity assembly, then we think unification should be fine overall. It’s not perfect, but it feels like the best way to keep folks from hitting this error.

@fabiocav and I would therefore propose:

1. Properly document the _FunctionsSkipCleanOutput and FunctionsPreservedDependencies options (Tracked by https://github.com/Azure/azure-functions-host/issues/8887)
2. Establish the auto-upgrade process for this assembly (Tracked by https://github.com/Azure/azure-functions-host/issues/8888)
3. Move this assembly into full unification (Tracked by https://github.com/Azure/azure-functions-host/issues/8889)
4. _Alongside any possible future major version change, we would plan to move this to the private assembly model. (Tracked by https://github.com/Azure/azure-functions-host/issues/8890)

This item could be closed once those first three items are complete. I will create separate work items and edit this post to include those tomorrow.

With runtime 4.11.2.19273 and Microsoft.NET.Sdk.Functions 4.1.3, you should be able to successfully reference up to 6.21.0 of Microsoft.IdentityModel.Tokens. I was able to succeed with that using the originally provided repro project.

That version is still behind latest public, and if I did update the package past that, I do indeed see the error as described. 6.21.0 is from ~4 months ago, and the latest (6.24.0) is from about 12 days ago.

Short term tactics

We have opened a PR to snap to latest (6.24.0): https://github.com/Azure/azure-functions-host/pull/8863

You will need to update to the latest Functions dependencies once that is fully in and a release is cut.

Fixing this longer-term

Eventually a new version of Microsoft.IdentityModel.Tokens will come out, and the host will be behind again, causing this issue to present until yet another host version can be released which absorbs that update. As has been mentioned here, FunctionsSkipCleanOutput is a workaround, but it’s not ideal.

We need to discuss the best way to fix this and will be doing so. One option on the table that @fabiocav and I were chatting about briefly today would be to add this to assembly unification. That means at runtime, your app would snap to whatever the host has. It would prevent this error from occurring and causing a crash, but it would also mean you that you aren’t actually getting the latest version you specify. There would still be a dependency on the host to continue moving this forward. To be clear, we would intend to keep updating it (within the major version range), but there is a question of how much lag time is acceptable, etc. We will be looking at process improvements regardless.

Assembly unification comes with some tradeoffs, and it’s not clear that’s the direction we’d go, but I wanted to capture that based on the conversation. We’ll continue discussing options - hopefully the tactical version update will help folks here for the time being while we do.

Same problem for me

Got the same issue. “Could not load file or assembly ‘Microsoft.IdentityModel.Tokens’” (version 6.16.0) even when the .csproj file contains a PackageReference.

.net6 / azure functions v4

Sounds like there is a fix on the way being tracked in issue https://github.com/AzureAD/microsoft-identity-web/issues/1548