helm: Unable to `helm install` from a private OCI container (401 Unauthorized error)

I am unable to install a chart from a private OCI registry, but I am able to helm push/pull to/from the private OCI registry.

echo $RegPassword | helm registry login $Server:5000 --username pacregistry --password-stdin
Login Succeeded
helm pull oci://$Server:5000/helm/nginx-ingress --version 0.10.1
Pulled: registry.server.io:5000/helm/nginx-ingress:0.10.1
Digest: sha256:194ae3c7a5927accf6f0944938b9f294b410280c36091f9770b903a1e47ba506
helm install oci://$Server:5000/helm/nginx-ingress --version 0.10.1 --generate-name
Error: INSTALLATION FAILED: failed to download "oci://registry.server.io:5000/helm/nginx-ingress" at version "0.10.1"

Debug output shows Unauthorized, however, as shown above the login has succeeded and push/pull works:

install.go:178: [debug] Original chart version: "0.10.1"
Error: INSTALLATION FAILED: unexpected status code [manifests 0.10.1]: 401 Unauthorized
helm.go:88: [debug] unexpected status code [manifests 0.10.1]: 401 Unauthorized
github.com/containerd/containerd/remotes/docker.(*dockerResolver).Resolve
	github.com/containerd/containerd@v1.5.4/remotes/docker/resolver.go:299
oras.land/oras-go/pkg/oras.Pull
	oras.land/oras-go@v0.4.0/pkg/oras/pull.go:45
helm.sh/helm/v3/internal/experimental/registry.(*Client).Pull
	helm.sh/helm/v3/internal/experimental/registry/client.go:235
helm.sh/helm/v3/pkg/getter.(*OCIGetter).get
	helm.sh/helm/v3/pkg/getter/ocigetter.go:57
helm.sh/helm/v3/pkg/getter.(*OCIGetter).Get
	helm.sh/helm/v3/pkg/getter/ocigetter.go:36
helm.sh/helm/v3/pkg/downloader.(*ChartDownloader).DownloadTo
	helm.sh/helm/v3/pkg/downloader/chart_downloader.go:99
helm.sh/helm/v3/pkg/action.(*ChartPathOptions).LocateChart
	helm.sh/helm/v3/pkg/action/install.go:753
main.runInstall
	helm.sh/helm/v3/cmd/helm/install.go:194
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:125
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.2.1/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
	runtime/proc.go:225
runtime.goexit
	runtime/asm_amd64.s:1371
INSTALLATION FAILED
main.newInstallCmd.func2
	helm.sh/helm/v3/cmd/helm/install.go:127
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.2.1/command.go:902
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
	runtime/proc.go:225
runtime.goexit
	runtime/asm_amd64.s:1371

Install from the local repo, not OCI registry, works deploying it to the kubernetes cluster:

helm install nginx-ingress/ --generate-name
W0913 01:14:29.244126  385267 warnings.go:70] networking.k8s.io/v1beta1 IngressClass is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 IngressClassList
W0913 01:14:29.488496  385267 warnings.go:70] networking.k8s.io/v1beta1 IngressClass is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 IngressClassList
NAME: nginx-ingress-1631520867
LAST DEPLOYED: Mon Sep 13 01:14:28 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The NGINX Ingress Controller has been installed.

Output of helm version:

helm version
version.BuildInfo{Version:"v3.7.0-rc.3", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.7"}

Output of kubectl version:

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2+k3s1", GitCommit:"5a67e8dc473f8945e8e181f6f0b0dbbc387f6fca", GitTreeState:"clean", BuildDate:"2021-08-09T21:42:04Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): Fedora 33

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 5
Comments: 15 (10 by maintainers)

Commits related to this issue

HELM_REGISTRY_CONFIG: change default file of registry config to config.json Changes default registry config file from `registry.json` to `config.json`. This aligns with the `config.json` that is wide... — committed to zonggen/helm by deleted user 2 years ago
HELM_REGISTRY_CONFIG: change default file of registry config to config.json Changes default registry config file from `registry.json` to `config.json`. This aligns with the `config.json` that is wide... — committed to zonggen/helm by deleted user 2 years ago
HELM_REGISTRY_CONFIG: change default file of registry config to config.json Changes default registry config file from `registry.json` to `config.json`. This aligns with the `config.json` that is wide... — committed to zonggen/helm by deleted user 2 years ago
HELM_REGISTRY_CONFIG: change default file of registry config to config.json Changes default registry config file from `registry.json` to `config.json`. This aligns with the `config.json` that is wide... — committed to zonggen/helm by deleted user 2 years ago
HELM_REGISTRY_CONFIG: change default file of registry config to config.json Changes default registry config file from `registry.json` to `config.json`. This aligns with the `config.json` that is wide... — committed to zak905/helm by deleted user 2 years ago

Most upvoted comments

On working on another project involving Docker credentials, the filename config.json is hardcoded everywhere in Docker codebase… It was a mistake to rename this file to registry.json. Should probably instead do something like /home/myname/.config/helm/.docker/config.json, which would allow people to use DOCKER_CONFIG=/home/myname/.config/helm/.docker

jdolitsky on Oct 4, 2021

same issue with AWS ECR and Helm 3.7.0

for now falling back to 3.6.3:

if you’d like to stick with 3.7.0, my workaround is :

helm pull oci://$oci_addr/$my-app --version $version
mkdir chart && tar -zxvf "$my-app-$version.tgz" -C chart
helm upgrade myrelease ./chart/$my-app -i [...]

totogtr on Sep 20, 2021

I think I’ve found the error. When calling helm install the registry.Client.credentialsFile is set to /home/myname/.cache/helm/registry/config.json while it should be /home/myname/.config/helm/registry.json as it is true for the helm pull command. Not sure though where it is set

Ka0o0 on Oct 4, 2021

Thanks for the detailed information. I’m going to promote this to a bug.

@jdolitsky can you please take a closer look?

bacongobbler on Sep 13, 2021

Digging just a little deeper, I am able to install, but only if I remove the private registry password, i.e. not use htpasswd and start up the private registry as anonymous. So it’s likely somewhere in the password decoding and passing that to the private registry where the issue is.

@zonggen since version v3.7.0 they are removing the helm chart <action> to fold it into helm and eventually remove ‘experimental’ described in more detail here: https://github.com/helm/helm/pull/9782

Thank you for verifying it doesn’t work also using the docs example as well.

hickersonj on Sep 13, 2021