helm: Tiller pods can't connect to k8s apiserver

@mattus Thanks a lot, i was stuck for ~ 3 days with this at work trying to deploy a k8s cluster. This should really be documented somewhere. What i did to solve the issue was:

• kubectl --namespace=kube-system edit deployment/tiller-deploy and changed automountServiceAccountToken to true. Then ‘helm list’ was giving me: Error: configmaps is forbidden: User “system:serviceaccount:kube-system:default” cannot list configmaps in the namespace “kube-system” That was fixed with solution from https://github.com/kubernetes/helm/issues/2687:
• kubectl --namespace=kube-system create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

FYI this is happening when I use terraform to build an AKS k8s cluster in Azure. This issue gave a clue: https://github.com/terraform-providers/terraform-provider-kubernetes/issues/38

Editing the deploy like @fossxplorer mentions seems to help me. Here’s a handy one-liner to do just that:

kubectl -n kube-system patch deployment tiller-deploy -p '{"spec": {"template": {"spec": {"automountServiceAccountToken": true}}}}'

Had something very similar to this, our eventual fix was to ensure automountServiceAccountToken on the ServiceAccount running tiller was set to true. We were creating the service account with terraform which was defaulting to false for reasons discussed in https://github.com/terraform-providers/terraform-provider-kubernetes/issues/38.

Manually creating the service account with kubectl does not set this value (it defaults to true when unset).

Alternatively, one is able to install with the default service account specified: helm init --service-account default which will install tiller to the kube-system namespace.

Just upgrade helm to version 3.

None of this stuff worked for me and I had this exact same error as 3870. The only thing I got to work was helm init --net-host which actually allowed Tiller to reach api-server via localhost in the Node’s network namespace.

For the record, this was done on a k8s server built by hand by cloning the k8s repo and building the binaries, not deployed by kubeadm or some other deployer; arguably, @bacongobbler’s comment about a “misconfigured kubernetes cluster” would likely apply in my case, but it’s a lab box and that is okay w/ me.

This has been fixed in helm v2.9.1, which is available here: https://github.com/kubernetes/helm/releases/v2.9.1

Thanks everyone!

@bacongobbler KUBERNETES_SERVICE_HOST exists in env indeed but it’s value is a cluster IP like 10.254.0.1. If I don’t set KUBERNETES_MASTER, tiller is unable to connect apiserver port 8080. Only if it is set to master physical address like 192.168.56.101, it’s ok.

So the problem may be tiller can’t get the correct apiserver address from k8s when tiller initializes the k8s apiserver connection.

This might help others that come upon this issue when working with terraform. The following terraform configuration was able to both create the service account and properly initialize tiller. The key is the override parameter to the helm provider.

data "google_client_config" "current" {}

provider "kubernetes" {
  load_config_file = false

  host                   = "https://${var.master_ip}"
  token                  = "${data.google_client_config.current.access_token}"
  cluster_ca_certificate = "${base64decode(var.cluster_ca_certificate)}"
}

provider "helm" {
  namespace       = "kube-system"
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  override        = ["spec.template.spec.automountserviceaccounttoken=true"]

  kubernetes {
    host                   = "https://${var.master_ip}"
    token                  = "${data.google_client_config.current.access_token}"
    cluster_ca_certificate = "${base64decode(var.cluster_ca_certificate)}"
  }
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller"
  }

  # TODO: give specific permissions
  role_ref {
    name = "cluster-admin"
    kind = "ClusterRole"
  }

  subject {
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }
}

did you try #4229? It resolved the issues in my case

I meet the same problem, it was due to KUBECONFIG is not default. so you shoud specify KUBECONFIG env to the right position.

I ran into this on a fresh GKE cluster that I manually enabled RBAC on per Google’s docs here.

I had already installed Helm, so things broke down when I enabled RBAC. After creating a Serviceaccount and ClusterRoleBinding for tiller, upgrading Helm with helm init --service-account tiller --upgrade was still producing the error above: Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp [::1]:8080: connect: connection refused.

As others have said, updating the tiller deployment to change automountServiceAccountToken: true fixed my issue.

editing tiller-deployment, spec.template.spec.automountServiceAccountToken and setting it to true (it was false) seemed to help.