azure-cli: Permission denied on msal_token_cache.bin.lockfile

Describe the bug

When running Terraform locally, I get an error originating from the Azure CLI. This happens on several versions, all relatively new, of the azurerm Terraform module. I am able to reproduce the error without going via Terraform, so I think that this issue ought to be filed in this repo.

Command Name az account get-access-token

Errors:

ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: [Errno 13] Permission denied: 'C:\\Users\\Per Stolpe\\.azure\\msal_token_cache.bin.lockfile'
Traceback (most recent call last):
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 657, in execute
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 691, in _run_job
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 328, in __call__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/profile/custom.py", line 74, in get_access_token      
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/_profile.py", line 381, in get_raw_token
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/_profile.py", line 588, in _create_credential
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/auth/identity.py", line 182, in get_user_credential
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/auth/msal_authentication.py", line 41, in __init__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msal/application.py", line 872, in get_accounts
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msal/application.py", line 910, in _find_msal_accounts
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msal_extensions/token_cache.py", line 53, in find
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\msal_extensions/cache_lock.py", line 29, in __enter__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\portalocker/utils.py", line 199, in __enter__
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\portalocker/utils.py", line 128, in acquire
  File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\portalocker/utils.py", line 175, in _get_fh
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\Per Stolpe\\.azure\\msal_token_cache.bin.lockfile'

To Reproduce:

The azurerm Terraform module fires off several commands in parallel, triggering this error. Since this resembles a race condition, I run below Powershell and wait until it says that something went wrong.

0..1000 | `
    Foreach-Object -Parallel {
        $subscriptionId = switch ($_ % 9) {
            0 { 'subscription id guid'; break }
            1 { 'subscription id guid'; break }
            2 { 'subscription id guid'; break }
            3 { 'subscription id guid'; break }
            4 { 'subscription id guid'; break }
            5 { 'subscription id guid'; break }
            6 { 'subscription id guid'; break }
            7 { 'subscription id guid'; break }
            8 { 'subscription id guid'; break }
        }

        az account get-access-token --subscription $subscriptionId
    }

Expected Behavior

Just the access token JSON printed a thousand times.

Environment Summary

Windows-10-10.0.19041-SP0
Python 3.8.9
Installer: MSI

azure-cli 2.30.0

Additional Context

When I first tried to recreate this issue using only four subscription ids, it never happened. Thus, it seems that an increasing amount of subscription ids increases the probability of triggering this issue. In my Terraform code, I call multiple Terraform modules, which themselves call other modules, so nine subscription ids is a realistic amount for me at least.

I should add that I have tried to reboot, uninstall and reinstall the CLI.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 23 (11 by maintainers)

Most upvoted comments

I can confirm that the updated version of the Azure CLI works perfectly for my “real” use-case with Terraform, which caused me to create this issue. Thank you and good job!

@stolpe, @kensykora , @devbeard , thanks for your patience. The issue has been fixed in Azure CLI 2.30+ 's upstream module, msal-extensions. Now, users of Azure CLI 2.30 and 2.31 can simply upgrade your environment like this:

"C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe" -m pip install -U --yes msal-extensions

We expect upcoming Azure CLI 2.32 will include this module by default.

(@jiasli , feel free to convert this issue to a Q&A, and mark this message as the final answer.)

@rayluo, expiresOn seems to be set two hours ahead for me, and it automatically refreshes using version 2.30.0 of the CLI and the feature branch of MSAL, described above.

I just installed a new VM with Azure CLI version 2.34.1 and I get the error occasionally now 😦 Can I downgrade to a specific version to fix this?

Can you share more on your usage scenario, such as: what kind of Azure CLI commands that you use, what is the VM’s OS, and the error trace you got, etc.? That could help us investigate and try to find the root cause.

By the way, you may consider create a new github issue for your case. I’m not sure whether interactions on a closed issue would still grab enough attention from @jiasli’s team.

You are right about creating a new issue. I will do that and thanks

I just installed a new VM with Azure CLI version 2.34.1 and I get the error occasionally now 😦 Can I downgrade to a specific version to fix this?

Can you share more on your usage scenario, such as: what kind of Azure CLI commands that you use, what is the VM’s OS, and the error trace you got, etc.? That could help us investigate and try to find the root cause.

By the way, you may consider create a new github issue for your case. I’m not sure whether interactions on a closed issue would still grab enough attention from @jiasli’s team.

Hi @rayluo,

When I installed the dev branch as instructed above, the error reappeared for me.

(I always had the workaround of using 2.29 of the CLI, so no worries about my environment)

Hi @jiasli,

I installed the feature branch according to your instructions above, and that did indeed solve the issue I had. Admittedly, the issue was far from easily reproducible with my repro repo, but with my Terraform configuration, it happened every time. Now, that too works flawlessly with version 2.30.0 of the Azure CLI.

Thank you very much for your help. You may close this issue as resolved when the time is right for that.

@Stolpe we are working on https://github.com/AzureAD/microsoft-authentication-extensions-for-python/pull/100 to refine concurrent read access of Azure CLI and MSAL.

Perhaps you can help install that feature branch skip-read-lock of msal-extensions and see if the issue is mitigated?

Open a PowerShell terminal with Administrator permission, then run:

& "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe" -m pip uninstall --yes msal-extensions
& "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe" -m pip install git+https://github.com/AzureAD/microsoft-authentication-extensions-for-python@skip-read-lock#egg=msal-extensions

This will install the feature branch skip-read-lock of msal-extensions to C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\msal_extensions.

Reference: https://pip.pypa.io/en/stable/topics/vcs-support/