terraform-provider-kubernetes: Can not set internal annotations to specify load balancer ssl certs

With the recent change of disallowing internal annotations it is no longer possible to setup internal annotations.

I want to use these internal annotations to provide SSL certificates/arn configuration for the load balancers. Any idea for a workaround?

Affected Resource(s)

service

Terraform Configuration Files

"aws_elb_hosted_zone_id" "main" {}

data "aws_acm_certificate" "hello" {
  domain   = "*.hello.com"
  statuses = ["ISSUED"]
}

resource "kubernetes_service" "hello" {
  metadata {
    name = "terraform-hello-integrated-example"

    annotations {
      "service.beta.kubernetes.io/aws-load-balancer-ssl-cert" = "${data.aws_acm_certificate.hello.arn}"
    }
  }

  spec {
    selector {
      app  = "hello"
    }

    port {
      port             = 80
      target_port = 80
    }

    type = "LoadBalancer"
  }
}

resource "aws_route53_record" "hello" {
  zone_id = "HELLO"
  name    = "hello"
  type    = "A"

  alias {
    name                   = "${kubernetes_service.hello.load_balancer_ingress.0.hostname}"
    zone_id                = "${data.aws_elb_hosted_zone_id.main.id}"
    evaluate_target_health = true
  }
}

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 20
Comments: 60 (25 by maintainers)

Most upvoted comments

Here’s another set of “internal” kubernetes.io annotations that are disallowed: https://github.com/kubernetes-sigs/aws-alb-ingress-controller/blob/master/docs/ingress-resources.md#annotations. This ingress controller is absolutely unusable without whitelisting the large list.

C’mon guys, the examples where this is blocking people for legitimate usage are far outweighing and far outnumbering the original reasons for blocking it, and severely degrades the usability of this provider. Please revert #50 or reduce its scope.

+15

mbarrien on Oct 30, 2018

This has now been released in 1.7.0: https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/CHANGELOG.md#170-may-22-2019 🎉 🚀

+12

tdmalone on May 23, 2019

I think that especially because of the reasoning behind #50 this is a big issue for many of us.

Unfortunately, the usage after this change showed that some tasks that were possible before are not possible anymore. For those of us who rely on these features (internal load balancers, aws certificates, …) this makes those things really hard do achieve.

Therefore, I would be really interested in feasible solution such as some whitelisting of certain annotations or similar.

Phylu on May 18, 2018

With terraform 0.12+, you need to specify annotations as a map and with an equal sign:

resource "kubernetes_secret" "my-secret" {
  metadata {
    name = "my-secret"
    annotations = {
      "kubernetes.io/service-account.name" = "my-account"
    }
  }
  type = "kubernetes.io/service-account-token"
}

pdecat on Jun 12, 2019

Haha, lack of support of internal annotations just made this and couple of other Kubernetes resources useless. Especially in enterprise. Thanks for removing this. Now automation of this just got much complicated.

jakubigla on Dec 10, 2018

Hi, same issue for me, as I need to specify the well known pv.beta.kubernetes.io/gid annotation on my persistent volumes.

I’ve read https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50#issue-251016641 and understand that the root issue from the terraform point of view is more with dynamic kubernetes annotations that potentially change all the time like those listed at https://kubernetes.io/docs/reference/labels-annotations-taints/ :

beta.kubernetes.io/arch
beta.kubernetes.io/os
kubernetes.io/hostname
beta.kubernetes.io/instance-type
failure-domain.beta.kubernetes.io/region
failure-domain.beta.kubernetes.io/zone

As maintaining the list of such annotations exhaustively can seem overwhelming if not impossible as time goes, what about allowing to whitelist annotations by configuration?

I guess doing so at the provider level would be practical:

provider "kubernetes" {
  internal_annotations_whitelist = [
    "pv.beta.kubernetes.io/gid",
    "service.beta.kubernetes.io/aws-load-balancer-internal",
    "service.beta.kubernetes.io/aws-load-balancer-ssl-cert",
 ]
}

What do you think?

PS: I’ll use the pod security context’s fs_group field to resolve for my current FS GID issue, but I believe whitelisting will come in handy anyway.

Edit: #325 is a better alternative as it does not require the user to actively maintain the whitelist.

pdecat on Apr 26, 2019

The same problem exists when your LB shall not have an external IP, e.g., in development environments with only internal IPs. I’d be happy if this issue can be solved. Thank you.

annotations {
   "service.beta.kubernetes.io/aws-load-balancer-internal" = "0.0.0.0/0"
}

frankthelen on Oct 5, 2017

And here’s a dodgy workaround (partial implementation - it provisions, but not updates or destroys). YMMV.

resource "null_resource" "eks_service_metadata_annotations" {
  provisioner "local-exec" {
    command = <<EOF
kubectl annotate --overwrite --namespace ${local.eks_namespace} \
  service ${kubernetes_service.sample.metadata.0.name} \
  'service.beta.kubernetes.io/aws-load-balancer-backend-protocol=http' \
  'service.beta.kubernetes.io/aws-load-balancer-ssl-cert=${local.acm_certificate_arn}'
EOF
  }
}

tdmalone on Feb 22, 2019

@dh-harald, the main issue I had with using https://github.com/sl1pm4t/terraform-provider-kubernetes was it’s lack of support for exec authentication (because it is using an older k8s client) which is needed to integrate with AWS (at least without hacks). I took your suggested changes and filed PR #244. I really don’t understand why #Hashicorp is allowing this provider to languish and become useless by both making changes that prevent its use in real-world scenarios and not adding support for current k8s objects like Ingress, ClusterRole, etc.

rpatrick00 on Dec 10, 2018

I’m not a go developer, so I won’t open a pull request, but here’s my solution for this problem: https://github.com/dh-harald/terraform-provider-kubernetes/commit/d0b48e8ec7e991f702aa6e0ea1dfdfef6380c509

Idea: @pdecat / Solution: @mb290

Here’s an example:

provider "kubernetes" {
  internal_annotations_whitelist = [
    "storageclass.kubernetes.io/is-default-class",
  ]
}

resource "kubernetes_storage_class" "gp2" {
  metadata {
    name = "gp2"

    annotations {
      "storageclass.kubernetes.io/is-default-class" = "true"
    }
  }
  storage_provisioner = "kubernetes.io/aws-ebs"
  parameters {
    type      = "gp2"
    encrypted = "true"
  }
}

I don’t know is it known, but if you’re using true instead of "true", terraform translates it into 1, and it won’t work

dh-harald on Nov 19, 2018

Now that development is starting back up on this, it would be great to have this revisited. I would love to use this in some different projects, but kube2iam used the internal annotations to allow pods to assume an IAM role.

andyvanosdale on Oct 30, 2018

I’m also now hit with the bug and having the list of approved annotations would seriously help. I’m not a developer and had never even ventured into Go before, but managed to piece together the following extension to the existing function validateAnnotations in https://github.com/terraform-providers/terraform-provider-kubernetes/blob/master/kubernetes/validators.go.

func validateAnnotations(value interface{}, key string) (ws []string, es []error) {
	m := value.(map[string]interface{})
	for k, _ := range m {
		errors := utilValidation.IsQualifiedName(strings.ToLower(k))
		if len(errors) > 0 {
			for _, e := range errors {
				es = append(es, fmt.Errorf("%s (%q) %s", key, k, e))
			}
		}

		permittedInternalKubAnnotations := map[string]bool {
			"service.beta.kubernetes.io/azure-load-balancer-internal": true,
			"service.beta.kubernetes.io/azure-load-balancer-resource-group": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-enabled": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name": true,
			"service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix": true,
			"service.beta.kubernetes.io/aws-load-balancer-backend-protocol": true,
			"service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled": true,
			"service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout": true,
			"service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout": true,
			"service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled": true,
			"service.beta.kubernetes.io/aws-load-balancer-internal": true,
			"service.beta.kubernetes.io/aws-load-balancer-proxy-protocol": true,
			"service.beta.kubernetes.io/aws-load-balancer-ssl-cert": true,
			"service.beta.kubernetes.io/aws-load-balancer-ssl-ports": true,
		}
		
		val := permittedInternalKubAnnotations[k];
	
		if isInternalKey(k) && val != true {
			es = append(es, fmt.Errorf("%s: %q is internal Kubernetes annotation", key, k))
		}
	}
	return
}

I’ve tested the above and it works with my specific use case of needing an internal load balancer in Azure for provisioning a POC. Whilst i understand why the check for internal providers was built in the first place, this is severely impacting the usefulness of the Kubernetes provider without the ability for this exception.

If there is appetite for me to submit a pull request i’m more than happy to but I suspect there is a significantly more efficient way of the above check 😃. I’ll also have to figure out the part of that stops the attempt to re-enable the load balancer every time as this is what happens when I re-run:

  ~ kubernetes_service.nginx
      metadata.0.annotations.%:                                                       "0" => "1"
      metadata.0.annotations.service.beta.kubernetes.io/azure-load-balancer-internal: "" => "true"

TechyMatt on Jun 12, 2018

Same here, I am trying to make storage class to be default. And I need storageclass.kubernetes.io/is-default-class annotation.

Using of this annotation is not restricted by Kubernetes and documented there https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

r-moiseev on Apr 18, 2018

By my opinion this check is unneeded, it’s up to user to not use these annotations, and use them when he knows what he is doing.

offlinehacker on Apr 9, 2018

So is there’s any progress with this? I mean it’s 2019 and April already hello? @pdecat @AlexBoulyga Anyone? Help? Devops can’t live without terraform!

holms on Apr 8, 2019

List of related issues/PRs:

tdmalone on Feb 22, 2019

where could I have found this in reference docs? For this specific issue, https://www.terraform.io/docs/providers/kubernetes/r/secret.html#annotations nor https://www.terraform.io/docs/configuration/syntax.html were that helpful.

That specific use case is missing from the docs. The most extensive source for examples are the acceptance test cases, e.g. https://github.com/terraform-providers/terraform-provider-kubernetes/blob/v1.7.0/kubernetes/resource_kubernetes_secret_test.go#L342 Maybe they could be extracted and put somewhere more easily discoverable.

Edit: I’ve noticed a few examples were outdated for terraform 0.12, working on a PR.

pdecat on Jun 12, 2019

Hi. Yes, a release is coming in the next couple of days. I’m aiming for it to include PodAffinity as well, so I’m looking into getting that ready as well.

On Fri 17. May 2019 at 19:46, Pavel Eremeev notifications@github.com wrote:

Can we have a release tagged with the fix included, please?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/terraform-providers/terraform-provider-kubernetes/issues/60?email_source=notifications&email_token=AAIL5G3552LIY2B2JR7P2R3PV3VQLA5CNFSM4DYLP6H2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVNG7I#issuecomment-493540221, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIL5G7ACG2BGC4GUM6OMI3PV3VQLANCNFSM4DYLP6HQ .

–

— Sent from my phone.

alexsomesan on May 17, 2019

Can we have a release tagged with the fix included, please?

selslack on May 17, 2019

I took a similar approach and used “az aks get-credential”. It works well enough, but I find I have to be a lot more explicit about my dependencies.

erick-thompson on Apr 19, 2019

Is this still stalled? I’m trying to use External DNS, and I’ve run into this issue. It’s very annoying. I don’t like this idea of terraform making decisions about what I can and can’t do in k8s. I’ve run into other issues like this (not being able to set automountServiceAccountToken on deployments).

Is there a workaround for this particular issue?

erick-thompson on Apr 16, 2019

@Xyrodileas, most of us have gone off the reservation and built our own version of the provider to get around this problem. I created PR #244 for the main provider to address the problem in the way suggested by one of the contributors. Others, like sl1pm4t / terraform-provider-kubernetes, have forked the code since Hashicorp seems uninterested in making this provider capable of real-world deployments. Either way, the only real workaround at this point is to either build your own custom provider or use the local-exec hack to run kubectl.

rpatrick00 on Apr 10, 2019

Also just run into this trying to set storage classes on persistent volume claims in OpenShift, without the ability to set set the annotations to specify a storage tier the claims will sit in pending and Terraform times out with “Still creating…”

rikkuness on Dec 21, 2018

No, I don’t hack anything… I just configure the provider:

provider "kubernetes" {
  host                   = "${aws_eks_cluster.main.endpoint}"
  token                  = "${data.external.aws_iam_authenticator.result["token"]}"             # https://github.com/hashicorp/terraform/issues/17173
  load_config_file       = false
  cluster_ca_certificate = "${base64decode(aws_eks_cluster.main.certificate_authority.0.data)}"
}

data "external" "aws_iam_authenticator" {
  program = ["sh", "-c", "AWS_PROFILE=${var.aws_account} aws-iam-authenticator token -i ${aws_eks_cluster.main.name} -r ${aws_iam_role.admin.arn} | jq -r -c .status"]
}

When you created the cluster first, you need to skip the -r (role) part, until you didn’t add the configmap for the role as well

dh-harald on Dec 10, 2018

Please read https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50#issue-251016641, which explains why this check exists.

bitglue on May 17, 2018

This is pretty annoying, why enforce this at all? Doesn’t kubernetes API prevent this?

bobbytables on Jan 10, 2018

Hi @udangel-r7 I’m sorry for any inconvenience caused by this.

Allowing users to specify internal (kubernetes.io) annotations is something I plan to look into eventually, but it’s non trivial for us to support. See full explanation at https://github.com/terraform-providers/terraform-provider-kubernetes/pull/50#issue-251016641

radeksimko on Sep 1, 2017