azure-cli: Installation of extension(s) is not working due to parsing error of extensionCommandTree.json: json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

Describe the bug

When trying to install extensions, just getting error

The command failed with an unexpected error. Here is the traceback:
Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

This happens both locally (using e.g. docker run -it --rm mcr.microsoft.com/azure-cli) and at Azure DevOps.

Related command

az account subscription list (even without logging in)

Errors

The command failed with an unexpected error. Here is the traceback:
Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

Issue script & Debug output

aba60738e9d4:/# az account subscription list
The command failed with an unexpected error. Here is the traceback:
Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/json/__init__.py", line 335, in loads
    raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 589, in execute
    parsed_args = self.parser.parse_args(args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/knack/parser.py", line 261, in parse_args
    return super().parse_args(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 1869, in parse_args
    args, argv = self.parse_known_args(args, namespace)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 1902, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 2117, in _parse_known_args
    stop_index = consume_positionals(start_index)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 2073, in consume_positionals
    take_action(action, args)
  File "/usr/local/lib/python3.11/argparse.py", line 1978, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/local/lib/python3.11/argparse.py", line 1241, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/parser.py", line 281, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 1902, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 2117, in _parse_known_args
    stop_index = consume_positionals(start_index)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 2073, in consume_positionals
    take_action(action, args)
  File "/usr/local/lib/python3.11/argparse.py", line 1962, in take_action
    argument_values = self._get_values(action, argument_strings)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/parser.py", line 272, in _get_values
    value = super(AzCliCommandParser, self)._get_values(action, arg_strings)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/argparse.py", line 2506, in _get_values
    self._check_value(action, value[0])
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/parser.py", line 300, in _check_value
    use_dynamic_install = try_install_extension(self, args)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/extension/dynamic_install.py", line 119, in try_install_extension
    _check_value_in_extensions(cli_ctx, parser, args, use_dynamic_install == 'yes_without_prompt')
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/extension/dynamic_install.py", line 150, in _check_value_in_extensions
    ext_name = _search_in_extension_commands(cli_ctx, command_str, allow_prefix_match=allow_prefix_match)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/extension/dynamic_install.py", line 87, in _search_in_extension_commands
    cmd_chain = _get_extension_command_tree(cli_ctx)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/azure/cli/core/extension/dynamic_install.py", line 38, in _get_extension_command_tree
    EXT_CMD_TREE.data = response.json()
                        ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)

Expected behavior

Extension installed fine and command that was asked done.

Environment Summary

{
  "azure-cli": "2.57.0",
  "azure-cli-core": "2.57.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {}
}

Additional context

No response

About this issue

  • Original URL
  • State: closed
  • Created 5 months ago
  • Reactions: 10
  • Comments: 24 (2 by maintainers)

Most upvoted comments

ℹ️ Official Announcement

We have identified the root cause to be the incorrect metadata of the Command Tree file:

https://github.com/Azure/azure-cli/blob/bad3331ef22549d4c67d2f9265cd44776c9d54b4/src/azure-cli-core/azure/cli/core/extension/dynamic_install.py#L29

We have already fixed it and dynamic installation should now work as expected. Please retry the commands.

Thanks for reporting this issue and apologize for the inconvenience caused.

It seems to be happening when az doesn’t recognise a command and is looking for an extension, as it happens for eg. az abc123. Manually installing extensions using az extension add --name {extension} appears to be a workaround.

This workaround works fine

I use the SSH extension in a pipeline, and explicitly doing az extension add --name ssh avoided the above-mentioned issue.

It seems to be happening when az doesn’t recognise a command and is looking for an extension, as it happens for eg. az abc123. Manually installing extensions using az extension add --name {extension} appears to be a workaround.

Confirmed, the error is now gone for me, without changing anything on my side.

We’ve experienced this issue on our side and running az account help --debug indicates that it indeed fails on https://azurecliextensionsync.blob.core.windows.net/cmd-index/extensionCommandTree.json

extensionCommandTree.json: Unicode text, UTF-8 (with BOM) text, with very long lines (65533), with no line terminators

However, I suspect the CLI code is not prepared for decoding JSON files with BOM.

az devops configure is failing for us

Everything seems okay now, just ran the same pipeline and it is executing without any problem

I’m also hitting this in Github Action, using the following job:

    uses: azure/CLI@v1
    with:
      azcliversion: 2.56.0
      inlineScript: |
        export AZURE_DEVOPS_EXT_PAT="${{ secrets.AZURE_ARTIFACTS_PAT }}"
        az config set extension.use_dynamic_install=yes_without_prompt
        az artifacts universal download --organization "$ORGANIZATION" --project "7ac49839-f0e7-4bcd-a5c3-6b465a2217e9" --scope project --feed "Common" --name "$NAME" --version "${{env.REQUESTED_FRONTEND_VERSION}}" --path $GITHUB_WORKSPACE/wwwroot

The same job was working fine until one hour before, then it just broke. The key “runs-on” is set to “ubuntu-latest”.

We are looking into this and will reply here after we figure out.

az devops configure is also failing for us and we are blocked because of that.

We’re also hitting this, our automation pipelines are all blocked. This is a new regression caused last night in Azure CLI, things were working fine last night for the same pipeline:

image

az aks pod-identity add is also not working.

Thank you for opening this issue, we will look into it.