helm: helm v3.6.1 breaks compatibility with azure container registry

Output of helm version:

version.BuildInfo{Version:"v3.6.1", GitCommit:"61d8e8c4a6f95540c15c6a65f36a6dd0a45e7a2f", GitTreeState:"clean", GoVersion:"go1.16.5"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"archive", BuildDate:"2021-05-14T14:09:09Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): AKS

Steps to reproduce:

add acr using azure-cli
update repo data
install/upgrade/pull chart from said repo

Expected outcome: chart can be installed/upgraded/pulled without issues

Actual outcome: adding repo and updating repo data works, everything else fails

$ az acr helm repo add --name myrepo
This command is implicitly deprecated because command group 'acr helm' is deprecated and will be removed in a future release. Use 'helm v3' instead.
"myrepo" has been added to your repositories

$ helm repo up
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "myrepo" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm upgrade --reuse-values --debug mychart myrepo/mychart
Error: failed to fetch https://myrepo.azurecr.io/helm/v1/repo/_blobs/mychart-1.5.41.tgz : 401 Unauthorized
helm.go:81: [debug] failed to fetch https://myrepo.azurecr.io/helm/v1/repo/_blobs/mychart-1.5.41.tgz : 401 Unauthorized
helm.sh/helm/v3/pkg/getter.(*HTTPGetter).get
	helm.sh/helm/v3/pkg/downloader/chart_downloader.go:99
helm.sh/helm/v3/pkg/action.(*ChartPathOptions).LocateChart
	helm.sh/helm/v3/pkg/action/install.go:704
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:130
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.1.3/command.go:852
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.1.3/command.go:960
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.1.3/command.go:897
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:80
runtime.main
	runtime/proc.go:225
runtime.goexit
	runtime/asm_amd64.s:1371

helm.sh/helm/v3/pkg/downloader.(*ChartDownloader).DownloadTo
	helm.sh/helm/v3/pkg/downloader/chart_downloader.go:99
helm.sh/helm/v3/pkg/action.(*ChartPathOptions).LocateChart
	helm.sh/helm/v3/pkg/action/install.go:704
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:130
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.1.3/command.go:852
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.1.3/command.go:960
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.1.3/command.go:897
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:80
runtime.main
	runtime/proc.go:225
runtime.goexit
	runtime/asm_amd64.s:1371

Steps to mitigate: do not upgrade to v3.6.1, use HelmToolinstaller in Azure pipeline to install oder version

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 20 (7 by maintainers)

Most upvoted comments

Wait, it wasn’t clear to me to use both --pass-credentials and --repo. Anyways, here’s a snippet to get rid of az acr helm repo add and the patching from my previous post:

REPO="myrepo"
#CRED=($(az acr credential show -n ${REPO} --query '{username: username, password: passwords[0].value}' -o tsv))
CRED=($(az acr credential show -n ${REPO} | jq -r '.username, .passwords[0].value'))
HOST=$(az acr show -n ${REPO} --query 'loginServer' -o tsv)
REPO_URL="https://${HOST}/helm/v1/repo"

helm repo add ${REPO} ${REPO_URL} \
    --username ${CRED[0]} \
    --password ${CRED[1]} \
    --pass-credentials

EDIT: single query for credentials EDIT 2: use jq for shorter code and direct extraction of values (azure-cli creates a new object)

josefschabasser on Jun 25, 2021

I’m happy to report that Helm 3.6.2 fixes parts of the issue. If both the repo URL and the chart URL both contain the port (https://…:443/…) then the download works as before Helm 3.6.1. Since 443 is the default port, few people will add their repos using urls including :443. I’m not sure how many repository implementations add the default port explicitly, but at least Artifactory does. As long as doing so is “legal”, I think helm might need to normalize the URLs before comparing, so that explicitly adding default ports to a URL does not break what is expected to work. Any thoughts?

erpel on Jun 30, 2021