terraform-provider-kubernetes: Unable to use kubernetes provider with fixed limited permissions - see here: https://github.com/hashicorp/terraform-provider-azurerm/pull/21229

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.4.4
Kubernetes provider version: 2.19.0
Kubernetes version: 1.24.9
Azurerm provicer: 3.51.0

Affected Resource(s)

Terraform Configuration Files

# Configure the Microsoft Azure Provider
provider "kubernetes" {
  host                   = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.host
  username               = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.username
  password               = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.password
  client_certificate     = base64decode(data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.client_certificate)
  client_key             = base64decode(data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.client_key)
  cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.cluster_ca_certificate)
}
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.51.0"
    }
    azuread = {
      source  = "hashicorp/azuread"
      version = ">= 2.36.0"
    }
    kubernetes = {
    source = "hashicorp/kubernetes"
    version = ">= 2.19.0"
    }
    helm = {
      source = "hashicorp/helm"
      version = "2.9.0"
    }
  }


  required_version = ">= 0.14.9"
  backend "azurerm" {
  }
}

data "azurerm_kubernetes_cluster" "aks_provider_config" {
  name                = var.env_config[var.ENV][ "aks_cluster_name" ]
  resource_group_name = var.env_config[var.ENV][ "aks_rg_name" ]
}

data "kubernetes_namespace_v1" "proj_ns" {
  metadata {
    name = local.proj_name
  }
}

Debug Output

Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: Unauthorized
│ 
│   with data.kubernetes_namespace_v1.proj_ns,
│   on var-proj.tf line 37, in data "kubernetes_namespace_v1" "proj_ns":
│   37: data "kubernetes_namespace_v1" "proj_ns" {
│ 
╵

Steps to Reproduce

See here: https://github.com/hashicorp/terraform-provider-azurerm/issues/21183

1. Create AKS Cluster with Azure AD auth with RBAC and local accounts enabled
2. Create a service principal
3. Assign the principal Azure Kubernetes Service Cluster User Role to allow fetch the limited permission kubeconfig
4. Assign the principal Azure Kubernetes Service RBAC Admin Role to a specific namespace
5. Authenticate terraform with the specific service principal and configure the k8s provider
6. Try to fetch the data.kubernetes_namespace_v1

Expected Behavior

Kubernetes resources should be able to be fetched via data and resources should be created according to the limited permissions in the specific namespace

Actual Behavior

Terraform returns with error “unauthorized”

Important Factoids

I did some testing and the outcome is, that the fetch of the limited permissions works now:

2023-04-11T13:21:45.761Z [DEBUG] provider.terraform-provider-azurerm_v3.51.0_x5: AzureRM Request: 
POST /subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.ContainerService/managedClusters/XXX/listClusterUserCredential?api-version=2023-02-02-preview HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19.3 (amd64-linux) go-autorest/v14.2.1 hashicorp/go-azure-sdk/managedclusters/2023-02-02-preview HashiCorp Terraform/1.4.4 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev VSTS_2c406b0a-3caf-4961-98e2-e310b237dd52_build_241_0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 0
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 16ef209b-5c71-1f06-9efe-412a949223cd
Accept-Encoding: gzip: timestamp=2023-04-11T13:21:45.761Z
2023-04-11T13:21:45.949Z [DEBUG] provider.terraform-provider-azurerm_v3.51.0_x5: AzureRM Response for https://management.azure.com/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.ContainerService/managedClusters/XXX/listClusterUserCredential?api-version=2023-02-02-preview: 
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 11 Apr 2023 13:21:44 GMT
Expires: -1
Pragma: no-cache
Server: nginx
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 16ef209b-5c71-1f06-9efe-412a949223cd
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1198
X-Ms-Request-Id: 09f91280-9e21-4b60-bb28-1871c7e4a1d2
X-Ms-Routing-Request-Id: WESTEUROPE:20230411T132145Z:c88433b2-eb1d-4cf0-a40d-9f4c3d36dbd7

{
  "kubeconfigs": [
   {
    "name": "clusterUser",
    "value": "XXX"
   }
  ]
 }: timestamp=2023-04-11T13:21:45.949Z

The base64 decoded kubeconf (the value) looks correct:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: XXX
    server: https://XXX.azmk8s.io:443
  name: XXX
contexts:
- context:
    cluster: XXX
    user: clusterUser_XXX
  name: XXX
current-context: XXX
kind: Config
preferences: {}
users:
- name: clusterUser_XXX
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --environment
      - AzurePublicCloud
      - --server-id
      - XXX
      - --client-id
      - XXX
      - --tenant-id
      - XXX
      - --login
      - devicecode
      command: kubelogin
      env: null
      provideClusterInfo: false

To debug, I tried to use this service principal sequence for kubectl. I use the following sequence:

az login --service-principal -u XXX -p XXX --tenant XXX

(This command fetches the identical kubeconfig as the terraform sequence)
az aks get-credentials --name XXX --resource-group XXX --overwrite-existing

(However, when I try to use `kubectl get all -n proj_ns` directly, i get the following:
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code XXX to authenticate.
It only works after I use kubelogin)
kubelogin convert-kubeconfig -l azurecli

After the kubelogin convert, the kubeconfig under .kube/config looks like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: XXX
    server: https://XXX.azmk8s.io:443
  name: XXX
contexts:
- context:
    cluster: XXX
    user: clusterUser_XXX
  name: XXX
current-context: XXX
kind: Config
preferences: {}
users:
- name: clusterUser_XXX
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - get-token
      - --login
      - azurecli
      - --server-id
      -  XXX
      command: kubelogin
      env: null
      provideClusterInfo: false

So I don’t know what runs behind the terraform curtain, but I suspect the kubelogin part of the steps is not accounted for, thus getting the “unauthorized” response because it does not get the token from the azurecli context.

References

• https://github.com/hashicorp/terraform-provider-kubernetes/issues/1964
• https://github.com/hashicorp/terraform-provider-azurerm/issues/21183
• https://github.com/hashicorp/terraform-provider-azurerm/pull/21229

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment