azure-cli: `apt-get update` for https://packages.microsoft.com/ randomly fails with "server certificate verification failed" on Ubuntu Trusty 14.04

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug Installing azure cli via the installation script or by running the manual installation fails at random. As part of our CI/CD process, we need to install the azure cli. With no apparent reason, sometimes it works and sometimes it does’t. When it fails, it fails with an error about the certificate of packages.microsoft.com

The following NEW packages will be installed:
  azure-cli
0 upgraded, 1 newly installed, 0 to remove and 249 not upgraded.
Need to get 63.1 MB of archives.
After this operation, 924 MB of additional disk space will be used.
Err https://packages.microsoft.com/repos/azure-cli/ trusty/main azure-cli all 2.27.2-1~trusty
  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
E: Failed to fetch https://packages.microsoft.com/repos/azure-cli/pool/main/a/azure-cli/azure-cli_2.27.2-1~trusty_all.deb  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I tried adding apt-get update --fix-missing. I tried adding trusted=yes in the configuration of the repo in the /etc/apt/sources.list.d/azure-cli.list file. And I tried running it with this script:

sudo apt-get update
sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg
curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg > /dev/null
echo -n | openssl s_client -showcerts -connect packages.microsoft.com:443 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /usr/local/share/ca-certificates/packages.microsoft.com.crt
update-ca-certificates --fresh
AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | sudo tee /etc/apt/sources.list.d/azure-cli.list
sudo apt-get update
sudo apt-get install azure-cli

But still no luck. It works most of the times, but when it doesn’t, it breaks the build, and I don’t know why it doesn’t work.

To Reproduce install az cli via script or manually https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

Expected behavior Install the az cli with no problems.

Environment summary Interactive script. Tried to reproduce it by connecting to the container with ssh and still most of the time it worked, but sometime it would still fail.

Additional context It appears to be a problem with the apt-get update command / A problem with packages.microsoft.com

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 30 (14 by maintainers)

Most upvoted comments

Summary

Root Cause

We recently updated the TLS/SSL certificate on half of the mirrors of packages.microsoft.com gradually in a rolling fashion. 14 mirrors with the new certificate have this issue, and the remaining 14 mirrors are good as of now as they still use the old valid certificate.

The issue with the new certificate seems to be caused by reversed order of the intermediate CA and root CA. This is causing problem in an older version of Linux (Ubuntu Trusty 14.04) and seems to be fine on newer version of Linux (Ubuntu Xenial 16.04, Bionic 18.04 and Focal 20.04). So the problem is isolated as of now to Ubuntu Trusty.

Solution

We were able to reproduce this issue in our Pre-Production Environment with a Trusty machine. We changed the order of certificate in our Pre-Production Environment and the issue seems to be resolved in Ubuntu Trusty.

We will roll out the fixed new certificate for impacted machines (14 mirrors out of 28) next week after the long weekend (Monday, September 6th is US Labor Day holiday).

Mitigation

In the meantime, the mitigation is to ask apt/apt-get to trust the packages.microsoft.com SSL certificate:

sudo apt-get -o Acquire::https::packages.microsoft.com::Verify-Peer="false" update
sudo apt-get -o Acquire::https::packages.microsoft.com::Verify-Peer="false" install azure-cli

@usternik, @nakazonot, @jshimazu, thank you so much for reporting this issue and all the valuable information provided!

jiasli on Sep 4, 2021

We have deployed the fixed new certificate to these regions:

Node	IP Address
csd-apt-sea-d-1.southeastasia.cloudapp.azure.com	52.230.121.169
csd-apt-sea-d-2.southeastasia.cloudapp.azure.com	20.188.102.6
csd-apt-eas-d-1.eastasia.cloudapp.azure.com	13.75.64.135
csd-apt-eas-d-2.eastasia.cloudapp.azure.com	65.52.183.205

I have tested these nodes with the script in https://github.com/Azure/azure-cli/issues/19405#issuecomment-912531180, and everything is working as expected.

jiasli on Sep 9, 2021

A summary of different systems and results of

openssl version
openssl s_client -showcerts -servername packages.microsoft.com -connect 52.230.121.169:443 </dev/null

Trusty 14.04 / OpenSSL 1.0.1f 6 Jan 2014 ❌
Xenial 16.04 / OpenSSL 1.0.2g 1 Mar 2016 ✔
Bionic 18.04 / OpenSSL 1.1.1 11 Sep 2018 ✔
Focal 20.04 / OpenSSL 1.1.1f 31 Mar 2020 ✔

Perhaps openssl 1.0.1f is too old to recognize incorrect cert order? There is no update for openssl on Trusty anymore:

# apt-get install --only-upgrade openssl
Reading package lists... Done
Building dependency tree
Reading state information... Done
openssl is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

jiasli on Sep 3, 2021

Thank you @jiasli, it worked fine.

nakazonot on Sep 14, 2021