helm: Kubernetes cluster unreachable -> dial tcp: i/o timeout

This issue started to occur very recently and since I’ve made couple of updates simultaneously, I am not sure what broke helm.

Output of helm version: -> Installed with brew

version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.19"}

Output of kubectl version: -> Installed with brew

Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.24.0

Cloud Provider/Platform: AKS, EKS

Laptop: MacOS (M1)

When I run kubectlcommands against the cluster, they work perfectly. However when I run helmcommands (such as helm list -A), I get the following error which I have never seen until a week ago:

Error: Kubernetes cluster unreachable: Get "https://aksnr1euwplatformd001-rgnr1euwplatformd001-2482af9b.hcp.westeurope.azmk8s.io:443/version": dial tcp: lookup aksnr1euwplatformd001-rgnr1euwplatformd001-2482af9b.hcp.westeurope.azmk8s.io: i/o timeout

-> both kubectl and helm are using the same ~/.kube/config.

I have uninstalled & reinstalled helm again. It also didn’t work…
The cluster is public (copying creds with az aks get-credentials and running kubectl locally works).
I have tried helm 3.9.0 and it also didn’t work.
According to the compatibility matrix, helm 3.x should work just fine with k8s v1.24.0 (Though I also tried with k8s v.1.23.5 and it failed again.
I have tried the same with EKS and issue occured again.

Last remark: I have fired up an EC2 (Ubuntu 18.04) and installed kubectl, azure-cli and helm. Tried helm list -A and it worked. Can it be a brew or a Mac M1 problem?

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 4
Comments: 28 (4 by maintainers)

Most upvoted comments

I was running into the same issue and discovered it was being caused by a weird IPv6 DNS record in my System Settings. These greyed out rules were set automatically (based on my network setup I think), but after manually setting just 192.168.1.254 as my sole DNS server my issues were resolved instantly. System Preferences -> Network -> Advanced -> DNS

This worked for me. Thanks!

Aldiuser on Sep 26, 2022

I was running into the same issue and discovered it was being caused by a weird IPv6 DNS record in my System Settings. These greyed out rules were set automatically (based on my network setup I think), but after manually setting just 192.168.1.254 as my sole DNS server my issues were resolved instantly. System Preferences -> Network -> Advanced -> DNS

rc65 on Sep 23, 2022

This appears to be an upstream Go issue.

https://github.com/golang/go/issues/52839

eddiezane on Oct 12, 2022

I’m also getting this issue, and have tried all steps above without success.

adammfrank on Feb 6, 2023

I also have the problems with Helm 3.9.4 on macOS Monterey (version 12.6) with M1 chip. Installed Helm with brew.

Aldiuser on Sep 19, 2022

@sriramkuau1 That network overloading & freezing the computer entirely, is exactly what I also have experienced…

Additionally, kubectl also have caused similar issues and I’ve uninstalled/reinstalled it per brew again. Then it started to work.

utr1903 on Sep 12, 2022