helm: Cannot perform helm init behind proxy

Output of helm version: v2.10.0-rc.1 (https://github.com/kubernetes/helm/releases/tag/v2.10.0-rc.1)

Output of kubectl version: v1.11.0

Cloud Provider/Platform (AKS, GKE, Minikube etc.): bare metal

Description: When checking the latest RC version (https://github.com/kubernetes/helm/releases/tag/v2.10.0-rc.1) I am unable to perform “helm init” behind proxy:

$ ./helm init --node-selectors "beta.kubernetes.io/os"=linux
Creating /home/ubuntu/.helm
Creating /home/ubuntu/.helm/repository
Creating /home/ubuntu/.helm/repository/cache
Creating /home/ubuntu/.helm/repository/local
Creating /home/ubuntu/.helm/plugins
Creating /home/ubuntu/.helm/starters
Creating /home/ubuntu/.helm/cache/archive
Creating /home/ubuntu/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Error: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp 216.58.214.80:443: connect: connection timed out

My environment is set properly for proxy:

$ export
...
declare -x HTTPS_PROXY="http://some-proxy:1234"
declare -x HTTP_PROXY="http://some-proxy:1234"
...
declare -x http_proxy="http://some-proxy:1234"
declare -x https_proxy="http://some-proxy:1234"

And downgrading to latest stable release v2.9.1 (https://github.com/kubernetes/helm/releases/tag/v2.9.1) solves the issue:

$ ./helm init --node-selectors "beta.kubernetes.io/os"=linux
Creating /home/ubuntu/.helm/repository/repositories.yaml
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com
Adding local repo with URL: http://127.0.0.1:8879/charts
$HELM_HOME has been configured at /home/ubuntu/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 11
  • Comments: 52 (15 by maintainers)

Commits related to this issue

Most upvoted comments

Problem still exists for 2.10.0:

# http_proxy=http://proxy.company.com:3128 https_proxy=http://proxy.company.com:3128 no_proxy=localhost,127.0.0.1,.int.company.com /usr/local/sbin/helm init --service-account tiller
Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Error: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp 172.217.19.208:443: connect: connection timed out

No issues for 2.9.1:

# http_proxy=http://proxy.company.com:3128 https_proxy=http://proxy.company.com:3128 no_proxy=localhost,127.0.0.1,.int.company.com /usr/local/sbin/helm init --service-account tiller
Creating /root/.helm 
Creating /root/.helm/repository 
Creating /root/.helm/repository/cache 
Creating /root/.helm/repository/local 
Creating /root/.helm/plugins 
Creating /root/.helm/starters 
Creating /root/.helm/cache/archive 
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /root/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!

Create PR #4579 which should fix this.

@tardich I can tell 2.9.1 does work. I suggest you stick to that version for your tests. Are you able to curl or wget https://kubernetes-charts.storage.googleapis.com/index.yaml ? If so, try:

helm init --skip-refresh --wait
helm repo remove stable
helm repo add --ca-file=/etc/ssl/certs/ca-bundle.crt stable https://kubernetes-charts.storage.googleapis.com --debug
helm repo update

or even try to copy manually the index.yaml (if you wget worked)

For reference, with 2.9.1 my Dockerfile contained: RUN helm init --client-only --wait With 2.10.0 the workaround is:

RUN helm init --client-only --skip-refresh --wait
 && helm repo remove stable
 && helm repo add --ca-file=/etc/ssl/certs/ca-bundle.crt stable https://kubernetes-charts.storage.googleapis.com
 && helm repo update

It’s a good question. Generally speaking, once we go forward with the next minor release we avoid cutting patch releases for prior releases, doubly so if there’s a workaround for the patch. That’s mostly done to reduce workload constraints on our end since we’re a small team maintaining the project. I’ll bring it up with the rest of the team and see what they think about cutting another release.

If anyone would be willing to try the v2.11.0-rc.2 release candidates for the fix, that would also be appreciated (and should not require the workaround). 😃

@marchowarth’s workaround also worked for me on MacOS. Assuming you have openssl installed, add the repo as follows: helm repo add --ca-file=/usr/local/etc/openssl/cert.pem stable https://kubernetes-charts.storage.googleapis.com

I had the same issue with v2.10.0, can confirm it isn’t on the v2.11.0-rc.2.

Perhaps a naive question, but isn’t this a v2.10 bug? Will it be included in the v2.10.1 (bugfixes) release?

I think the reason for this is 3eaa1bfd3b040cb644c7974d32047dc70037cf95. This removed the use of http.DefaultClient and created the client with a new transport - this removed the default setting for using a proxy (http.ProxyFromEnvironment).

$ helm init --skip-refresh Creating /home/xxxxxx/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /home/xxxxxx/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure ‘allow unauthenticated users’ policy. To prevent this, run helm init with the --tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation Happy Helming! $ helm list Error: configmaps is forbidden: User “system:serviceaccount:kube-system:default” cannot list configmaps in the namespace “kube-system” $ helm repo add foo https://kubernetes-charts.storage.googleapis.com --debug Error: Looks like “https://kubernetes-charts.storage.googleapis.com” is not a valid chart repository or cannot be reached: Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp 172.217.24.208:443: connect: connection refused

Notice that our corp policy prevent any direct http connection to the internet, and all web traffic must be directed to the proxy. However it seems that helm repo command doesn’t respect the proxy settings.

@bacongobbler it is likely, that helm simply connected directly with the internet and ignored the proxy settings. In a real corporate environment, this is not possible, and though it fails.

I am not sure on how to reproduce this without the real need for a proxy. But I am happy to try anything you like on my machine with the problematic Helm version on it. I just need directions on which logs to capture…

maybe it would also make sense to scan the diff between the two versions, maybe the error is obvious for somebody familiar with the codebase?