azure-cli-extensions: Dataprotection ModuleNotFoundError 'azure.mgmt.resourcegraph'

I understand that of course I could just install this missing module via pip3 as a hotfix. But, I also think this is not how this extension should behave in the first place. Is there maybe something missing during installation of the extension? Didn’t experience any ModuleNotFoundErrors before. Find below the ‘az feedback --verbose’ output with the list of my installed extensions, I manually noted which of them is a preview version.

Extension name (the extension in question)

dataprotection

Description of issue (in as much detail as possible)

ModuleNotFoundError: No module named ‘azure.mgmt.resourcegraph’
command: az dataprotection backup-instance --help

This is autogenerated. Please review and update as needed.

Describe the bug

az feedback --verbose

Command Name az dataprotection backup-instance

Errors:

The command failed with an unexpected error. Here is the traceback:
No module named 'azure.mgmt.resourcegraph'
Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 583, in execute
    parsed_args = self.parser.parse_args(args)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/parser.py", line 261, in parse_args
    return super().parse_args(args)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1821, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 284, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1854, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2045, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2022, in consume_positionals
    take_action(action, args)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1931, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1210, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 284, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1854, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2045, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2022, in consume_positionals
    take_action(action, args)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1931, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1210, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 284, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1854, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2063, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2003, in consume_optional
    take_action(action, args, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1931, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1095, in __call__
    parser.print_help()
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2551, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 195, in format_help
    super(AzCliCommandParser, self).format_help()
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/parser.py", line 248, in format_help
    self.cli_help.show_help(self.prog.split()[0],
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/_help.py", line 165, in show_help
    else self.group_help_cls(self, delimiters, parser)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/help.py", line 253, in __init__
    child.load(options)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/_help.py", line 303, in load
    loader.versioned_load(self, options)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/_help_loaders.py", line 154, in versioned_load
    super(CliHelpFile, help_obj).load(parser)  # pylint:disable=bad-super-call
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/help.py", line 198, in load
    description = getattr(options, 'description', None)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/parser.py", line 241, in __getattribute__
    self.description = self._description() \
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 133, in description_loader
    op = self.get_op_handler(self.op_path)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
    handler = import_module(mod_to_import)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/NJ62KD/.azure/cliextensions/dataprotection/azext_dataprotection/custom.py", line 20, in <module>
    raise e
  File "/Users/NJ62KD/.azure/cliextensions/dataprotection/azext_dataprotection/custom.py", line 15, in <module>
    from .manual.custom import *  # noqa: F403
  File "/Users/NJ62KD/.azure/cliextensions/dataprotection/azext_dataprotection/manual/custom.py", line 17, in <module>
    from azure.mgmt.resourcegraph.models import \
ModuleNotFoundError: No module named 'azure.mgmt.resourcegraph'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

install extension via first use (implicit) or via az extension add -n dataprotection (explicit). Both ways led to the same outcome.
az dataprotection backup-instance -h or any az dataprotection backup-instance subcommand.

Expected Behavior

Shows help output.

Environment Summary

macOS-12.2.1-x86_64-i386-64bit, Darwin 21.3.0
Python 3.10.1
Installer: HOMEBREW

azure-cli 2.32.0 *

Extensions:
front-door 1.0.16
alertsmanagement 0.1.1
logic 0.1.5 (manual note: preview)
resource-graph 2.1.0
sentinel 0.1.1
ssh 1.0.0 (manual note: preview)
dataprotection 0.2.0
application-insights 0.1.14 (manual note: preview)

Dependencies:
msal 1.16.0
azure-mgmt-resource 20.0.0

Additional Context

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (11 by maintainers)

Most upvoted comments

This problem also happens in a virtual env, but not Azure CLI installed with MSI or DEB:

I tested with a sample script:

import sys

import azure
print(azure, azure.__path__)

sys.path.append(r'C:\Users\username\.azure\cliextensions\dataprotection')

import azure
print(azure, azure.__path__)

import azure.mgmt
print(azure.mgmt, azure.mgmt.__path__)

import azure.mgmt.resourcegraph
print(azure.mgmt.resourcegraph, azure.mgmt.resourcegraph.__path__)

System python:

> & "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe" D:\cli\testproj\main.py
<module 'azure' (namespace)> _NamespacePath(['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\lib\\site-packages\\azure'])
<module 'azure' (namespace)> _NamespacePath(['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\lib\\site-packages\\azure'])
<module 'azure.mgmt' (namespace)> _NamespacePath(['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\lib\\site-packages\\azure\\mgmt', 'C:\\Users\\username\\.azure\\cliextensions\\dataprotection\\azure\\mgmt'])
<module 'azure.mgmt.resourcegraph' from 'C:\\Users\\username\\.azure\\cliextensions\\dataprotection\\azure\\mgmt\\resourcegraph\\__init__.py'> ['C:\\Users\\username\\.azure\\cliextensions\\dataprotection\\azure\\mgmt\\resourcegraph']

Virtual env python:

> & "D:\cli\py310\Scripts\python.exe" D:\cli\testproj\main.py
<module 'azure' from 'd:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure\\__init__.py'> ['d:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure', 'D:\\cli\\py310\\lib\\site-packages\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-core\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-testsdk\\azure']
<module 'azure' from 'd:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure\\__init__.py'> ['d:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure', 'D:\\cli\\py310\\lib\\site-packages\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-core\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-testsdk\\azure']
<module 'azure.mgmt' (<_frozen_importlib_external._NamespaceLoader object at 0x0000025B0E2DA7A0>)> _NamespacePath(['D:\\cli\\py310\\lib\\site-packages\\azure\\mgmt'])
Traceback (most recent call last):
  File "D:\cli\testproj\main.py", line 14, in <module>
    import azure.mgmt.resourcegraph
ModuleNotFoundError: No module named 'azure.mgmt.resourcegraph'

In a virtual env which installs Azure CLI from source code, under azure and azure/mgmt folders, there are __init__.py files containing one line:

__path__ = __import__("pkgutil").extend_path(__path__, __name__)

These files make azure and azure.mgmt normal (non-namespace) packages, preventing Python from looking up additional library folders.

jiasli on Mar 31, 2022

Cause

Even though https://github.com/Azure/azure-cli/pull/13163 added the support to load azure.mgmt from extension installation folder:

https://github.com/Azure/azure-cli/blob/e2f55ec7de8ff0b19c05e082d408992ac1a1139c/src/azure-cli-core/azure/cli/core/extension/operations.py#L501-L521

def add_extension_to_path(extension_name, ext_dir=None):
    ext_dir = ext_dir or get_extension(extension_name).path
    sys.path.append(ext_dir)
    # If this path update should have made a new "azure" module available,
    # extend the existing module with its path. This allows extensions to
    # include (or depend on) Azure SDK modules that are not yet part of
    # the CLI. This applies to both the "azure" and "azure.mgmt" namespaces,
    # but ensures that modules installed by the CLI take priority.
    azure_dir = os.path.join(ext_dir, "azure")
    if os.path.isdir(azure_dir):
        import azure
        azure.__path__.append(azure_dir)
        azure_mgmt_dir = os.path.join(azure_dir, "mgmt")
        if os.path.isdir(azure_mgmt_dir):
            try:
                # Should have been imported already, so this will be quick
                import azure.mgmt
            except ImportError:
                pass
            else:
                azure.mgmt.__path__.append(azure_mgmt_dir)

it is not used in core’s extension loading process (but in other modules like network):

https://github.com/Azure/azure-cli/blob/af0c73a37ca3f540cb0b3f0b18e0f362932ab191/src/azure-cli-core/azure/cli/core/__init__.py#L351

                    ext_dir = ext.path or get_extension_path(ext_name)
                    sys.path.append(ext_dir)

We can show what azure really is at this time:

import azure
print(azure.__path__)

['d:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure', 'D:\\cli\\py310\\lib\\site-packages\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-core\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-testsdk\\azure']

Because azure has already been loaded, adding ext_dir won’t affect existing azure module.

Solution

The common practice is to put SDKs in a vendored_sdks folder, instead of installing from PyPI.

To my surprise, this extension was added by #3459 (Jun 3, 2021) but this bug is never discovered.

jiasli on Mar 31, 2022

Let me clarify 2 things:

python3’s import won’t work because Azure CLI runs in an isolated virtual env, meaning it doesn’t share system’s python packages.
Each python3’s execution is a separate process, so even you can do import with python3, this import statement only affects the current python3 process, so it doesn’t affect other python3 executions like az dataprotection backup-instance.

I am also able to reproduce this issue on Windows:

> az extension add -n dataprotection
> az dataprotection backup-instance -h
...
  File "C:\Users\name\.azure\cliextensions\dataprotection\azext_dataprotection\manual\custom.py", line 17, in <module>
    from azure.mgmt.resourcegraph.models import \
ModuleNotFoundError: No module named 'azure.mgmt.resourcegraph'

jiasli on Mar 31, 2022