azure-cli: SSL: CERTIFICATE_VERIFY_FAILED error on "az bicep install" Resurfaced

Describe the bug This is a reopening of #19420 the az bicep install command is once again throwing an SSL: CERTIFICATE_VERIFY_FAILED error

To Reproduce

$ az bicep install
Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

With debug:

$ az bicep install --debug
cli.knack.cli: Command arguments: ['bicep', 'install', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f603acf9b70>, <function OutputProducer.on_global_arguments at 0x7f603ac2b730>, <function CLIQuery.on_global_arguments at 0x7f603ac4b840>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'bicep': ['azure.cli.command_modules.resource']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: resource                  0.010        40       187
cli.azure.cli.core: Total (1)                 0.010        40       187
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 40 groups, 187 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : bicep install
cli.azure.cli.core: Command table: bicep install
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f603a76c598>]
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f603a725620>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x7f603a6af840>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f603a6af8c8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f603a6af9d8>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f603ac2b7b8>, <function CLIQuery.handle_query_parameter at 0x7f603ac4b8c8>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x7f603a6af7b8>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f603a6af950>]
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception:
cli.azure.cli.core.util: Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.6/site-packages/urllib3/connection.py", line 426, in connect
    tls_in_tls=tls_in_tls,
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 450, in ssl_wrap_socket
    sock, context, tls_in_tls, server_hostname=server_hostname
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/az/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/opt/az/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/opt/az/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/az/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/opt/az/lib/python3.6/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 156, in get_bicep_latest_release_tag
    response = requests.get("https://aka.ms/BicepLatestRelease", verify=ca_file)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/custom.py", line 3588, in install_bicep_cli
    ensure_bicep_installation(release_tag=version, target_platform=target_platform)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 106, in ensure_bicep_installation
    release_tag = release_tag if release_tag else get_bicep_latest_release_tag()
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/resource/_bicep.py", line 160, in get_bicep_latest_release_tag
    raise ClientRequestError(f"Error while attempting to retrieve the latest Bicep version: {err}.")
azure.cli.core.azclierror.ClientRequestError: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).

cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
az_command_data_logger: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)).
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f603a76c7b8>]
cli.__main__: Command ran in 0.447 seconds (init: 0.089, invoke: 0.358)

Expected behavior Bicep successfully installs.

Environment summary

Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

azure-cli 2.31.0

Extensions:
ml 2.0.3

Additional context Like the pervious ticket I am also behind a corporate proxy, other commands that require the proxy certificate like az login are working as expected. The REQUESTS_CA_BUNDLE environment variable has been set to point to a file including the proxy certificate. In addition, it doesn’t not appear that bicep is obeying the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable as running the following command export AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1 before attempting to do the install is having no effect.

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 20 (2 by maintainers)

Commits related to this issue

[ARM] Fix #20842: `az bicep`: Fix to use requests environment variables for CA bundle — committed to Azure/azure-cli by wwmoraes 2 years ago

Most upvoted comments

Upon inspecting the previous commit for #19420, the issue is that the bicep module is hardcoded to use the default cacert.pem certificate bundle (which is provided by the certifi python library dependency) when invoking HTTP requests via the requests python library. This approach ignores the REQUESTS_CA_BUNDLE environment variable that many of us are setting to utilize the az cli behind a corporate proxy. I believe these lines need to be adjusted to match how other az cli modules are setting the verify parameter for their calls to requests.get().

@Zerthick and @Jaffacakes82 - in the meantime, I found another workaround that allowed az bicep install to complete successfully for me on a Windows machine:

Append your corporate proxy certificate(s) to the end of the certifi library’s cacert.pem file. You’ll likely find the file at one of the following locations:

On Windows: C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem
On Linux: /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

jtrivett on Jan 24, 2022

Upon inspecting the previous commit for #19420, the issue is that the bicep module is hardcoded to use the default cacert.pem certificate bundle (which is provided by the certifi python library dependency) when invoking HTTP requests via the requests python library. This approach ignores the REQUESTS_CA_BUNDLE environment variable that many of us are setting to utilize the az cli behind a corporate proxy. I believe these lines need to be adjusted to match how other az cli modules are setting the verify parameter for their calls to requests.get().

@Zerthick and @Jaffacakes82 - in the meantime, I found another workaround that allowed az bicep install to complete successfully for me on a Windows machine:

Append your corporate proxy certificate(s) to the end of the certify library’s cacert.pem file. You’ll likely find the file at one of the following locations:

On Windows: C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem

On Linux: /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

Thanks for this - worked for me. In case it helps others: I’m on a mac and found the correct cacert.pem location by running az --version which spit out

Python location '/Users/<username>/.brew/Cellar/azure-cli/2.32.0/libexec/bin/python'

and subsequently found the right cacert.pem file here: ~/.brew/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/certifi/cacert.pem. Appending my corporate certs onto it allowed me to successfully run az bicep install 👍

steve-sellers on Jan 24, 2022

Thanks @ssvalkyrie & @jtrivett, worked a treat. @navba-MSFT, agree with @Zerthick, this needs addressing by Microsoft.

Jaffacakes82 on Mar 12, 2022

These suggestions work for us and provide a temporary workaround, for anyone looking to run Bicep from a Docker container (such as VS devcontainers) we use the following solution (with the cert fix that @jtrivett mentions)

FROM ubuntu
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        gnupg && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Get Azure CLI
# https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-apt?view=azure-cli-latest
RUN wget -q -O - https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/trusted.gpg.d/microsoft.asc.gpg > /dev/null && \
    echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ bionic main" | tee /etc/apt/sources.list.d/azure-cli.list && \
    apt-get update && \
    apt-get install -y --no-install-recommends azure-cli && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# Azure Bicep
RUN curl -Lo bicep https://github.com/Azure/bicep/releases/latest/download/bicep-linux-x64 && \
    chmod +x ./bicep && \
    mv ./bicep /usr/local/bin/bicep && \
    bicep --help

# Hack to fix Bicep cert issue: https://github.com/Azure/azure-cli/issues/20842#issuecomment-1018139759
RUN cp /etc/ssl/certs/<CORPORATE CERT CHAIN>.pem /opt/az/lib/python3.6/site-packages/certifi/cacert.pem

baileydoestech on Feb 11, 2022

Adding the Service team to look into this issue and share the update on this.

@josephkwchan, @jennyhunter-msft could you please provide an update on this once you get a chance ? Awaiting your reply.

navba-MSFT on Mar 25, 2022

@navba-MSFT Yeah, let us know what is the plan. As mentioned before there is a lot of people behind a corporate firewall.

MiguelElGallo on Mar 25, 2022

@navba-MSFT The suggestions above do indeed appear to work, though as others on this ticket have commented they are a workaround, bicep should be ideally obeying the REQUESTS_CA_BUNDLE environment variable to handle including corporate certificates and the AZURE_CLI_DISABLE_CONNECTION_VERIFICATION environment variable to disable the ssl check completely.

Zerthick on Mar 8, 2022

Also experiencing this in version 2.32.0 behind a corporate proxy.

Jaffacakes82 on Jan 13, 2022