terraform-provider-helm: Tiller does not install correctly

@Jeeppler, there are couple possibly unecessary bits here, but I left them in anyway.

1. automount_service_account_token = true should only be necessary for helm provider version < 0.7.0
2. tiller_image = "gcr.io/kubernetes-helm/tiller:v2.11.0" since this doesn’t seem to have been updated. Whoops. We should do that for 2.12.0/0.8.0.

TL;DR: yes, the service account needs to be created first.

provider "kubernetes" {
  version = ">= 1.4.0"
  ...
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "terraform-tiller"
    namespace = "kube-system"
  }

  automount_service_account_token = true
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "terraform-tiller"
  }

  role_ref {
    kind      = "ClusterRole"
    name      = "cluster-admin"
    api_group = "rbac.authorization.k8s.io"
  }

  subject {
    kind = "ServiceAccount"
    name = "terraform-tiller"

    api_group = ""
    namespace = "kube-system"
  }
}

provider "helm" "cluster_helm" {
  version = "~> 0.7.0"

  kubernetes {
    ...
  }

  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  namespace       = "${kubernetes_service_account.tiller.metadata.0.namespace}"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"
}

@jhoblitt Do I understand correctly that the main point of https://github.com/lsst-sqre/terraform-tinfoil-tiller is that it only listens on localhost?

I think the problem some people are having here is it seems like just including this block doesn’t actually install tiller because terraform doesn’t know to do anything to the remote k8 system just from a provider block.

provider "helm" {
  install_tiller  = true
  service_account = "${kubernetes_service_account.tiller.metadata.0.name}"
  tiller_image    = "${var.tiller_image}"
}

However if you include this block it looks like tiller does get installed (although terraform never really tells you it does)

resource "helm_release" "mydatabase" {
  name  = "mydatabase"
  chart = "stable/mariadb"

  set {
    name  = "mariadbUser"
    value = "foo"
  }

  set {
    name  = "mardiadbPassword"
    value = "qux"
  }
}

This is kinda a weird setup as I’ve never seen a terraform provider block actually install anything remotely. I think a better setup would be to remove the “install_tiller” argument and create a resource that installs tiller instead. Then if you try to use the helm provider and tiller isn’t installed just throw an error.

just my 2 cents.

Any dates when this will be fixed?

This will be fixed by #143

Just confirming that this seems to work when using Terraform 0.1.11, Helm Provider 0.7.0 and Helm 2.12.1 with the instructions provided by @Stelminator.

In case it’s useful to anyone else coming across this: I’d previously got this working by shell-ing out to helm init and I had to manually clear some old state referring to helm_releases using helm state before I could successfully do a terraform plan using the method described above.

Has anyone got this to work - I’m running 0.7.0, but tiller is not installed if I use the following definition. Is there anything else I need to set?

provider "helm" {
  install_tiller  = true
  namespace       = "kube-system"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"

  kubernetes {
    config_path = "${var.kube_config_path}"
  }
}

It hasn’t yet been updated for helm 2.11.0, but I hacked up a simple module for installing tiller with rbac (and without tiller being accessible to other pods).

https://github.com/lsst-sqre/terraform-tinfoil-tiller

@Jeeppler or just manually set the flag…

kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"automountServiceAccountToken":true}}}}'

This is kinda a weird setup as I’ve never seen a terraform provider block actually install anything remotely.

@cwiggs It is indeed weird. The plan is to deprecate that as soon as we have a tiller resource that meets all the expectations. See https://github.com/terraform-providers/terraform-provider-helm/pull/203#issuecomment-474933490

I traced this to what seems to be the most recent related change to helm: https://github.com/helm/helm/pull/4589 so, I think @olib963 is correct that 2.11.0+ should do the trick.

Hey @shamsalmon I had a similar problem, I think the issue is that tiller isn’t automounting the token for the service account, there is a PR open that will hopefully fix this soon by upgrading to helm 2.11.

https://github.com/terraform-providers/terraform-provider-helm/issues/122 might be related. Sorry if it is a different issue.

[EDIT] Looks like the problem was due to a duplicate declaration of helm provider, in a nested module. I also separated in two separate terraform “states” (to be applied sequentially) the ServiceAccount creation from the actual helm install, this to avoid the possible issue of the install job being executed before the ServiceAccout creation.

Same issue with AKS, my configuration: azurerm v 1.22.1 kubernetes v1.5.2 helm 0.8.0

 resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }

  automount_service_account_token = true
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller-binding"
  }

  subject {
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }

  role_ref {
    kind      = "ClusterRole"
    name      = "cluster-admin"
    api_group = "rbac.authorization.k8s.io"
  }
}

provider "helm" {
  
  kubernetes {
    host = "${azurerm_kubernetes_cluster.main.kube_config.0.host}"

    client_certificate     = "${base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_certificate)}"
    client_key             = "${base64decode(azurerm_kubernetes_cluster.main.kube_config.0.client_key)}"
    cluster_ca_certificate = "${base64decode(azurerm_kubernetes_cluster.main.kube_config.0.cluster_ca_certificate)}"
  }

  install_tiller  = "true"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.13.0"
}

Tiller account is actually created and tiller itself does get installed but with wrong serviceaccount (default instead of tiller) and wrong version (v2.11.0 instead of specified 2.13.0).

@Stelminator Using your config, I still cannot install Tiller:

  + module.habito-eks.kubernetes_cluster_role_binding.tiller
      id:                              <computed>
      metadata.#:                      "1"
      metadata.0.generation:           <computed>
      metadata.0.name:                 "terraform-tiller"
      metadata.0.resource_version:     <computed>
      metadata.0.self_link:            <computed>
      metadata.0.uid:                  <computed>
      role_ref.%:                      "3"
      role_ref.api_group:              "rbac.authorization.k8s.io"
      role_ref.kind:                   "ClusterRole"
      role_ref.name:                   "cluster-admin"
      subject.#:                       "1"
      subject.0.kind:                  "ServiceAccount"
      subject.0.name:                  "terraform-tiller"
      subject.0.namespace:             "kube-system"

  + module.habito-eks.kubernetes_service_account.tiller
      id:                              <computed>
      automount_service_account_token: "true"
      default_secret_name:             <computed>
      metadata.#:                      "1"
      metadata.0.generation:           <computed>
      metadata.0.name:                 "terraform-tiller"
      metadata.0.namespace:            "kube-system"
      metadata.0.resource_version:     <computed>
      metadata.0.self_link:            <computed>
      metadata.0.uid:                  <computed>

  + module.monitoring.helm_release.prometheus-operator
      id:                              <computed>
      chart:                           "stable/prometheus-operator"
      disable_webhooks:                "false"
      force_update:                    "false"
      keyring:                         "/home/tomislav/.gnupg/pubring.gpg"
      metadata.#:                      <computed>
      name:                            "prometheus-operator"
      namespace:                       "default"
      recreate_pods:                   "false"
      reuse:                           "false"
      reuse_values:                    "false"
      timeout:                         "300"
      verify:                          "false"
      version:                         "3.0.0"
      wait:                            "true"


Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.habito-eks.kubernetes_service_account.tiller: Creating...
  automount_service_account_token: "" => "true"
  default_secret_name:             "" => "<computed>"
  metadata.#:                      "" => "1"
  metadata.0.generation:           "" => "<computed>"
  metadata.0.name:                 "" => "terraform-tiller"
  metadata.0.namespace:            "" => "kube-system"
  metadata.0.resource_version:     "" => "<computed>"
  metadata.0.self_link:            "" => "<computed>"
  metadata.0.uid:                  "" => "<computed>"
module.habito-eks.kubernetes_cluster_role_binding.tiller: Creating...
  metadata.#:                  "" => "1"
  metadata.0.generation:       "" => "<computed>"
  metadata.0.name:             "" => "terraform-tiller"
  metadata.0.resource_version: "" => "<computed>"
  metadata.0.self_link:        "" => "<computed>"
  metadata.0.uid:              "" => "<computed>"
  role_ref.%:                  "" => "3"
  role_ref.api_group:          "" => "rbac.authorization.k8s.io"
  role_ref.kind:               "" => "ClusterRole"
  role_ref.name:               "" => "cluster-admin"
  subject.#:                   "" => "1"
  subject.0.kind:              "" => "ServiceAccount"
  subject.0.name:              "" => "terraform-tiller"
  subject.0.namespace:         "" => "kube-system"

Error: Error applying plan:

2 error(s) occurred:

* module.habito-eks.kubernetes_service_account.tiller: 1 error(s) occurred:

* kubernetes_service_account.tiller: Post http://localhost/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:80: connect: connection refused
* module.habito-eks.kubernetes_cluster_role_binding.tiller: 1 error(s) occurred:

* kubernetes_cluster_role_binding.tiller: Post http://localhost/apis/rbac.authorization.k8s.io/v1/clusterrolebindings: dial tcp 127.0.0.1:80: connect: connection refused

Using terraform v0.11.11

@Stelminator That’s correct. It avoids the need to configure x509 certs, network policy, etc.

I’ve updated https://github.com/lsst-sqre/terraform-tinfoil-tiller to work with 0.7.0. Eg.,

module "tiller" {
  source = "git::https://github.com/lsst-sqre/terraform-tinfoil-tiller.git//?ref=master"

  namespace       = "kube-system"
  service_account = "tiller"
  tiller_image    = "gcr.io/kubernetes-helm/tiller:v2.11.0"
}

provider "helm" {
  version = "~> 0.7.0"

  debug           = true
  service_account = "${module.tiller.service_account}"
  namespace       = "${module.tiller.namespace}"
  install_tiller  = false

  kubernetes {
    ...
  }
}

I tried with 0.7.0 but it still doesn’t work. I hope I am not missing any configuration