terraform-provider-kubernetes: Data resource for Service Account fails to find default token

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v0.13.3
Kubernetes provider version: 1.13.2
Kubernetes version: 1.16.11

Affected Resource(s)

data.kubernetes_service_account

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.
data "kubernetes_service_account" "test_sa" {
  metadata {
    name      = "test"
    namespace = "test"
  }
}

Debug Output

2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Response Details:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ RESPONSE ]--------------------------------------
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: HTTP/2.0 200 OK
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Length: 326
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Audit-Id: ...
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Cache-Control: no-cache, private
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Type: application/json
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Date: Mon, 04 Jan 2021 13:34:13 GMT
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "kind": "ServiceAccount",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "apiVersion": "v1",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "metadata": {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "name": "test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "selfLink": "/api/v1/namespaces/test/serviceaccounts/test",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "uid": "1ce457f7-276e-4579-a7df-ab489ae1c9cc",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "resourceVersion": "544195689",
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "creationTimestamp": "2020-06-05T05:51:48Z"
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "secrets": [
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   {
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "name": "test-token-ncwqf"
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   }
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  ]
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: }
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.700-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.702-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Request Details:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ REQUEST ]---------------------------------------
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: GET /api/v1/namespaces/test/secrets/test-token-ncwqf HTTP/1.1
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Host: k8s
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: User-Agent: HashiCorp/1.0 Terraform/0.13.3
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept: application/json, */*
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Authorization: Bearer ...
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Accept-Encoding: gzip
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.703-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Kubernetes API Response Details:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: ---[ RESPONSE ]--------------------------------------
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: HTTP/2.0 200 OK
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Length: 3113
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Audit-Id: ...
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Cache-Control: no-cache, private
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Content-Type: application/json
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: Date: Mon, 04 Jan 2021 13:34:13 GMT
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "kind": "Secret",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "apiVersion": "v1",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "metadata": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "name": "test-token-ncwqf",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "test",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "selfLink": "/api/v1/namespaces/test/secrets/test-token-ncwqf",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "uid": "baef6c8c-e549-4962-9e3c-eb0a9de64e6c",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "resourceVersion": "544195687",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "creationTimestamp": "2020-10-04T00:57:08Z",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "annotations": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "kubernetes.io/service-account.name": "test",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:    "kubernetes.io/service-account.uid": "1ce457f7-276e-4579-a7df-ab489ae1c9cc"
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   }
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "data": {
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "ca.crt": "...",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "namespace": "...",
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:   "token": "..."
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  },
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:  "type": "kubernetes.io/service-account-token"
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: }
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4:
2021-01-04T05:34:13.855-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: -----------------------------------------------------
2021-01-04T05:34:13.856-0800 [DEBUG] plugin.terraform-provider-kubernetes_v1.13.2_x4: 2021/01/04 05:34:13 [DEBUG] Skipping test-token-ncwqf as it wasn't created at the same time as the service account
2021/01/04 05:34:13 [ERROR] eval: *terraform.evalReadDataRefresh, err: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one
2021/01/04 05:34:13 [ERROR] eval: *terraform.EvalSequence, err: Failed to discover the default service account token: Unable to find any service accounts tokens which could have been the default one

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

Create a Service Account
Rotate Service Account Default Token
terraform apply apply fails because of differing creation timestamps

Expected Behavior

A valid service token should be found regardless of creation timestamp drift.

Actual Behavior

When a service account default token gets rotated, the new secret has a different timestamp and Terraform is unable to find the default token.

References

GH-848

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 10
Comments: 20 (6 by maintainers)

Commits related to this issue

Increased sa and secret creation_time difference check Increased service_account and secret creation_time difference check to a bigger limit, 60 seconds; instead of 3 seconds. If we have mutation web... — committed to amsgeodis/terraform-provider-kubernetes by amsgeodis 3 years ago
Increased sa and secret creation_time difference check (#1) Increased service_account and secret creation_time difference check to a bigger limit, 60 seconds; instead of 3 seconds. If we have mutatio... — committed to amsgeodis/terraform-provider-kubernetes by amsgeodis 3 years ago

Most upvoted comments

The problem is with the 3 seconds check performed in the below line.

https://github.com/hashicorp/terraform-provider-kubernetes/blob/e6ae58f9b75369c84965a7a13e67f0450e9aa65c/kubernetes/resource_kubernetes_service_account.go#L189

Can we have it increased to a bigger limit, may be 60 seconds? If we have mutation webhooks, the secret creation can take longer.

amsgeodis on Feb 11, 2021

Because import has to deal with the fact that there may be many secrets associated with the service account and it needs to discover which the ‘default’ one was - which the create operation has already defined as the secret that was created alongside the service account.

I would argue that the default_secret_name attribute should be removed and the secret list also be computed, but that would be a breaking change and I’ve not contributed to this provider in years.

wjam on Mar 3, 2021

Thx @rossdotpink because of that link i checked my nodes for clock drift… and indeed one of them was off just enough to cause the problem. Easy fix for me with my current issue… but i still think this should be more than 3 seconds… or at least a parameter, if time is still needed to get the correct token… which still seems a little “hacky” to me 🤷

erSitzt on Oct 11, 2021

I am also stuck with this. Seems like the motivation for 3 seconds was not entirely random, but I also really need this PR merged so I can get past this issue.

rossmobi on Sep 1, 2021

Note that the PR you linked to was just for adding support to import a service account and nothing more

@wjam Doesn’t it impact datasource?

amsgeodis on Mar 3, 2021

Why not use the secrets list from the sa object itself? My token got rotated and is month older then the sa.

wojtek-oledzki on Mar 3, 2021