Umbraco-CMS: Backoffice anti-forgery token can't be decrypted after deployments
Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)
v11-v13 (latest)
Bug summary
We’ve noticed lately that there’s some kind of issue with the backoffice anti-forgery token in some scenarios like app pool recycles and app pool switches causing the backoffice to not work and the server to return HTTP Status: 417 The anti-forgery token could not be decrypted.
This happens for us during deployments and has surfaced because customers have worked on a page that was loaded before our deployment. Then, due to the issue, they could not save their work. The only error indication is the X in the save button (no notification) so sometimes editors think that they have saved.
For the record: We do store the key ring on disk (using AddDataProtection()) and the issue does not affect anti-forgery tokens for the front end at all, only the backoffice. This also works for the Auth-cookie so we do not have to log in again in any of the scenarios.
On v12-v13 we consistently get “HTTP Status: 417 The anti-forgery token could not be decrypted” after app pool recycle on the production machines configured with default App Pool settings.
Specifics
The symptoms indicate that there is something special about how the key ring for the backoffice anti-forgery token is handled, almost as if it does not use the storage configured by ´AddDataProtection()` but creates something that seems to be dependent on the App Pool / App Pool Identity.
The anti forgery token on the front end works in all scenarios below, for both app pool recycle and app pool switches. I’ve also tested a vanilla MVC site with a form+anti forgery token without being able to reproduce the problem, this indicates that the backoffice token is handled in a different way causing the issue.
There are two types of actions that we’ve managed to find that would trigger this issue:
- App Pool recycle in certain scenarios, not all the time a recyle would invalidate the backoffice anti forgery token.
- App Pool switch, that is having two identical App Pools and switching the Websites App Pool between when the token was created and when the HTTP Post request is sent. This forum thread seems to be talking about something similar that happens with Azure Slot Swap.
Also, to be clear this does not happen:
- When running via Kesterl (dotnet run)
- When running via IIS Express from Visual Studio
We’ve tested on some of our production environments to identify if there were some changes in the CMS that might give us insights. This is the results:
| Machine | Recycle | App Pool Switch | Umb-version | Runtime | App-Pool Identity |
|---|---|---|---|---|---|
| Windows Server 2019 #1 | Works | 9.3.1 | 6.0.4 | ApplicationPoolIdentity | |
| Windows Server 2022 #1 | Works | 10.3.2 | 6.0.10 | ApplicationPoolIdentity | |
| Windows Server 2022 #2 | Works | Error | 11.2.1 | 7.0.11 | ApplicationPoolIdentity |
| Windows Server 2019 #2 | Works | 11.2.2 | 7.0.14 | ApplicationPoolIdentity | |
| Windows 2019 #1 | Works | Error | 11.3.1 | 7.0.5 | ApplicationPoolIdentity |
| Windows 2019 #1 | Works | 12.0.1 | 7.0.5 | ApplicationPoolIdentity | |
| Windows Server 2022 #2 | Error | 12.3.5 | 7.0.11 | ApplicationPoolIdentity | |
| Windows Server 2022 #2 | Error | 12.3.5 | 7.0.11 | ApplicationPoolIdentity | |
| Windows Server 2022 #2 | Error | 13.11.1 | 8.0.4 | ApplicationPoolIdentity | |
| Windows Server 2019 #2 | Error | 13.2.2 | 8.0.0 | ApplicationPoolIdentity |
Noticing that recycle worked in our tests up until 12.3.5 which might indicate that changes introduced to facilitate the content delivery API impacted this?
The “app pool switch” seems to have been an issue at least back to v 11.2.1.
Here is a screen share demonstrating the issue:
- First triggering the issue in the backoffice
- Demonstrating that the front end validates the anti forgery token
- Demonstrating that the front end anti forgery token works after the recyle, indicating that the issue is isolated to the backoffice token.
Steps to reproduce
I’ve created a zip for the project that I use for testing: https://www.dropbox.com/scl/fi/bd6wcnm3v19v6f6js4b1v/share-Umb-417-issue-MyProject.zip?rlkey=u1e2ezbzgqsmar30tow1vhx5f&dl=0
Here are the basic steps:
1. Create a new Umbraco website
dotnet new install Umbraco.Templates::13.2.2 --force && dotnet new sln --name "MySolution" && dotnet new umbraco --force -n "MyProject" --friendly-name "Administrator" --email "admin@example.com" --password "1234567890" --development-database-type SQLite && dotnet sln add "MyProject" && dotnet add "MyProject" package clean && dotnet run --project "MyProject"
2. Build a standard “MVC” form using a surface controller
This is to test the difference between the front-end anti-forgery token and the one used in the backoffice.
3. Publish the project
In the same folder as the .csproj, publish the project
dotnet publish
4. Setup IIS
In IIS:
- Create two identical App Pools (e.g. test-pool1 and test-pool2).
- Create a website and point the publish folder (bin/release/net8.0/publish) of the project, use App Pool “test-pool1”.
(Notice: In some scenarios, the recycle error is triggered when running the App Pool as ApplicationPoolIdentity and sometimes we’ve had to change the App Pool identity to a local user to trigger the problem).
5.1. Replicate “App Pool Recycle error”
- Run the website and login to the backoffice, open a page and press “save and publish”, notice that this works and returns a HTTP status of 200.
- Keep the browser open, recycle the app pool
- Without refreshing the browser, press “Save and publish” - notice the 417 error.
5.2. Replicate “App Pool Switch error”
- Run the website and login to the backoffice, open a page and press “save and publish”, notice that this works and returns a HTTP status of 200.
- Keep the browser open, in IIS: Open the website properties and switch App Pool from “test-pool1” to “test-pool2”.
- Without refreshing the browser, press “Save and publish” - notice the 417 error.
6 Front end
Perform step 5.1 and 5.2 with the form on the front-end to notice that the error is not present here.
Expected result / actual result
The backoffice anti-forgery token should be decrypted even after app pool recycle or app pool switch, without having to reload the backoffice page.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 15 (9 by maintainers)
Okey, I think I’ve figured out the problem.
The
Umbraco.Cms.Web.BackOffice.Security.BackOfficeAntiforgeryclass contains some interesting hacks =DA custom
ServiceCollectionis created to get around some issues with creating a custom AntiForgerty service, the fact that this DI container only exists inside this class means that any configuration made during startup will be ignored. This actually resonates with my initial findings that the backoffice anti forgery token does not use the key store configured withAddDataProtection.I figured that we could probably pass on the global configuration into the class-isolated DI container to be able to respect the data protection configuration.
I’ve created a simple work around that can be applied to any site to replace the current implementation:
Then during service registration, after
AddUmbraco():This implementation will have to add a new ctor (to get the “real” DI container) so I guess it would be considered a breaking change, maybe if someone from HQ want to have a look at my proposed implementation (that solves the problems for us) and let me know if you would like to have a PR =D
@enkelmedia Thanks so much! PatchedBackofficeAntiforgery approach solved the Backoffice anti-forgery issues on our production balance loader. We upgraded from Umbraco8 to Umbraco 13.5.1
@seanrockster The data protection stuff is not a ”Umbraco-thing”, its a .NET Core concept.
The keys used to encrypt/decrypt the cookies will be stored on the local machine by default, when you do a slot wrap there is a new machine, hence new keys that can’t decrypt your members cookies.
You need to persist the keys somewhere so that the new machine can use the same keys.
Have a look at the link from my previous answer.
Cheers!
Hi!
Thanks for looking at this!
I’ve been running my workaround in production for a while and it works fine, no problems since that update.
Let’s establish a common “vocabulary” for the different parts here =D
BackofficeAntiforgeryclass.In the current state of the backoffice, the backoffice auth cookies are encrypted using the global configuration and only the anti forgery token uses this local IOC that basically ignores any configuration made during startup.
Personally I would just find it confusing to have to configure data protection two times. I’m guessing that the vast majority of sites would use the same data protection configuration for the frontend and the backoffice. However if it would be implemented in a way so that it reuses the “global configuration” but also make it possible to set backoffice-specific settings if needed that sounds like a good solution to me.
There are plenty of examples out there about how to configure data protection so it would probably be good one could just follow them to “make it work” and then if someone needs to fiddle with different settings for the backoffice they could dive into the Umbraco docs to find out how.
Cheers!