terraform-provider-kubectl: failed to fetch resource from kubernetes: the server could not find the requested resource

The issue

I’m running into an error with karpenter yaml templates and I’m unsure on what the cause is. I’ve used the kubectl_manifest in the past and it worked fine on consecutive applies, but for some reason it’s not working with these custom resources.

EKS version: 1.27 gavinbunney kubectl version: 1.14 terraform: 1.4.6

Sample YAML file:

apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: system-infra
spec:
  instanceProfile: my_cluster-worker-role
  securityGroupSelector:
    kubernetes.io/cluster/my_cluster: owned
  subnetSelector:
    karpenter.sh/discovery: eks-private
  tags:
    Name: node
    managedBy: Terraform
    project: Sandbox
    repository: ..........
    stack: stack/sandbox
    workspace: default

When I try to apply it through terraform it will plan just fine, but when I try to apply it, it gives me the following error:

╷
│ Error: permissionsets failed to fetch resource from kubernetes: the server could not find the requested resource
│ 
│   with module.core_system.kubectl_manifest.permissions-sets[0],
│   on ......./rbac-addon.tf line 123, in resource "kubectl_manifest" "permissions-sets":
│  123: resource "kubectl_manifest" "permissions-sets" {
│ 
╵

I don’t know where to start looking to resolve this. The items never show up in the statefile, but the objects are created inside the cluster. It looks like it created them and tries to find them, but then doesn’t seem to find them?

I’ll keep digging as this provider is my only way of applying yamls to the cluster through terraform without using null_resource ( dirty last resort hack imo ) or the kubernetes_manifest ( which doesn’t work if the CRDs don’t exist ).

I’m not sure if this is related to the fact these are custom resources ( through CRDs ).

About this issue

  • Original URL
  • State: open
  • Created a year ago
  • Reactions: 29
  • Comments: 41 (3 by maintainers)

Commits related to this issue

Most upvoted comments

released in 2.0.2 (please raise an issue in case there are side effects, tested on kind from 1.19 to 1.27)

+39
alekc on Jul 19, 2023

Seems to be a generic issue for 1.27 - @gavinbunney any change you could look into this? Downgrading is not an options for many clusters…

+26
pat-s on Jun 15, 2023

Nice work guys!

I was going to suggest to maybe write a quick tutorial here on how to compile your fork but I have checked it and see that you’re already providing binary releases and the module is easily available under:

terraform {
  required_version = ">= 0.13"

  required_providers {
    kubectl = {
      source  = "alekc/kubectl"
      version = ">= 2.0.2"
    }
  }

On that note, I’m personally considering staying on the helm_release resource workaround.

It got me thinking after this incident that such a pretty popular provider, referenced by AWS terraform modules, can be abandoned and one day stop working completely.

If not now then in two years Kubernetes releases v1.38 or so, and everything will go south again.

+13
Oliniusz on Jul 19, 2023

i am seeing same issue on EKS kubernetes 1.27 with karpenter 0.27.5 and kubectl provider 1.14.0

+11
koertkuipers on Jun 12, 2023

For anybody stuck, blocked, losing hope for a progress on that - if you are in need of deploying your CRs before CRDs are available then I would suggest to do what I’ve done for now, eg. before:

resource "kubectl_manifest" "envoyfilter-proxy_protocol-internal" {
  yaml_body = <<-EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: EnvoyFilter
    metadata:
      name: proxy-protocol-internal
      namespace: istio-system
    spec:
      configPatches:
      - applyTo: LISTENER
        patch:
          operation: MERGE
          value:
            listener_filters:
            - name: envoy.filters.listener.proxy_protocol
            - name: envoy.filters.listener.tls_inspector
      workloadSelector:
        labels:
          istio: ingressgateway-internal
    EOF
  depends_on = [
    helm_release.istio-istiod
  ]
}

after:

resource "helm_release" "envoyfilter-proxy-protocol-internal" {
  name       = "envoyfilter-proxy-protocol-internal"
  namespace  = kubernetes_namespace.istio-system.metadata[0].name
  repository = "https://bedag.github.io/helm-charts/"
  chart      = "raw"
  version    = "2.0.0"
  values = [
    <<-EOF
    resources:
      - apiVersion: networking.istio.io/v1alpha3
        kind: EnvoyFilter
        metadata:
          name: proxy-protocol-internal
          namespace: istio-system
        spec:
          configPatches:
          - applyTo: LISTENER
            patch:
              operation: MERGE
              value:
                listener_filters:
                - name: envoy.filters.listener.proxy_protocol
                - name: envoy.filters.listener.tls_inspector
          workloadSelector:
            labels:
              istio: ingressgateway-internal
    EOF
  ]
  depends_on = [
    helm_release.istio-istiod,
    kubernetes_namespace.istio-ingress
  ]
}
+5
Oliniusz on Jul 7, 2023

This seems to be a bug due to the way the provider handles discovery. I haven’t investigated what exact behaviour in the api server changed in 1.27, but setting the GVK in a different way (see this commit) seems to fix it.

I’ve just opened https://github.com/alekc/terraform-provider-kubectl/pull/13 but on the @alekc’s fork, since that’s what I’ve been using for the past couple of months due to the low maintainer activity on this repo.

+4
valorl on Jul 19, 2023

I believe you also can’t create e.g. the EKS cluster from a scratch and apply your manifests in the same terraform step with kubernetes_manifest - that was the initial reason I switched to this kubectl_manifest.

Let’s be realistic though - this project has been abandoned. It would be awesome if @gavinbunney had time once more to update the code or add some other guys as maintainers but that might be unlikely.

+4
Oliniusz on Jun 30, 2023

https://github.com/gavinbunney/terraform-provider-kubectl/issues/270#issuecomment-1625610825 works great, thanks a ton!

+3
pat-s on Jul 8, 2023

More debugging ensued, switching to the kubernetes_manifest resource kinda figured there is a delta between what is being applied from the yaml and what the kube control plane returns the object as. For example

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.karpenter_provisioner_amd64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.requirements: new element 6 has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.karpenter_provisioner_arm64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.requirements: new element 6 has appeared.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Checking the resource yaml on the cluster and reconciling it back to the code fixed the issue on the kubernetes_manifest provider.

Another example

╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.runners-amd64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.template.spec.resources.requests["cpu"]: was cty.StringVal("1.6"), but now cty.StringVal("1600m").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.runners-amd64, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: .object.spec.template.spec.resources.limits["cpu"]: was cty.StringVal("4.0"), but now cty.StringVal("4").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Suggest trying the same with your resources and analyzing, or you could switch to the kubernetes_manifest resource.

+3
tejisin on Jun 16, 2023

Yup, pretty much my setup. Kubectl provider for initial karpenter manifests -> Helm release for ArgoCD -> argocd app manifests with kubectl for everything else.

+2
alekc on Oct 2, 2023

@alekc thanks a lot.

For the record I also had to rename the provider from state :

terraform state replace-provider registry.terraform.io/gavinbunney/kubectl registry.terraform.io/alekc/kubectl

+2
jodem on Sep 28, 2023

A dirty workaround that works

locals {
  karpenter_aws_node_template_yaml = <<EOF
    apiVersion: karpenter.k8s.aws/v1alpha1
    kind: AWSNodeTemplate
    metadata:
      name: ${local.karpenter_provisioner_name}
    spec:
      blockDeviceMappings:
      - deviceName: /dev/xvda
        ebs:
          deleteOnTermination: ${var.karpenter_provisioner_ebs_delete_on_termination}
          volumeSize: ${var.karpenter_provisioner_ebs_size}
          volumeType: ${var.karpenter_provisioner_ebs_type}
      subnetSelector:
        karpenter.sh/discovery: ${module.eks.cluster_name}
      securityGroupSelector:
        karpenter.sh/discovery: ${module.eks.cluster_name}
      tags:
        karpenter.sh/discovery: ${module.eks.cluster_name}
  EOF
}

resource "null_resource" "install_karpenter_aws_node_template" {
  triggers = {
    timestamp = timestamp()
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command     = "aws eks --region ${var.region} update-kubeconfig --name ${var.cluster_name} && kubectl apply -f - <<EOF\n${local.karpenter_aws_node_template_yaml}\nEOF"
  }
  depends_on = [helm_release.karpenter]
}
+2
vainkop on Jul 7, 2023

I’ve tried to create a fork and update the go modules. While this worked eventually to build some binaries after some minor code changes, the subsequent terraform apply unfortunately errored with issues I also get from the hashicorp kubectl provider (like “kind” unknown, “group” issues, etc.).

I am not versed enough to continue from here onwards but it might be that a simple update of the underlying modules/libraries is not enough to reinstate the previous behavior of this provider here, given all the upstream changes in the k8s API.

+2
pat-s on Jul 4, 2023

I have same issue trying to apply EniConfig resource on EKS cluster version 1.27. It worked on first apply but failed after that.

+2
duclm2609 on Jun 18, 2023

This repository has been abandoned and no PR will get approved by the maintainer.

+1
Oliniusz on Nov 29, 2023

@jodem I use the helm provider to install Flux and still use the kubectl provider to apply Flux CRs. The official kubernetes provider is terrible at CRDs, unfortunately. It requires cluster access to plan, making a single-pass cluster creation+bootstrap impossible.

+1
valorl on Oct 2, 2023

@jodem but you do need a running flux instance isn’t it? So you are still out of luck for all the tasks like bootstrapping etc. In theory you could make a use of a https://artifacthub.io/packages/helm/kiwigrid/any-resource helm chart, but it still cranky (and I had my share of. issues with helm provider so I tend to avoid it).

P.s. this provider is not actively maintained anymore, the one on my fork is 😉

+1
alekc on Oct 2, 2023

@seungmun yep, that’s the one. I’m not sure if that’s going to be universally true but my needs are pretty straightforward.

+1
karmajunkie on Jun 30, 2023

@seungmun per @tejisin’s comment I switched all my kubectl_manifest resources to kubernetes_manifest which works fine. Some of my resources needed to be redeployed IIRC, but in my case that wasn’t an issue.

+1
karmajunkie on Jun 30, 2023

Since everyone in this issue seems to be talking about EKS, I’d like to add that we’ve run into the same issue with kubeadm based clusters on GCP. So it’s definitely a 1.27 thing and not platform specific.

+1
itspngu on Jun 28, 2023

My EKS 1.27 also has the issue related to the provider and karpenter:

│ Error: default failed to fetch resource from kubernetes: the server could not find the requested resource
│
│   with module.karpenter.kubectl_manifest.karpenter_provisioner,
│   on ../../../../tf_modules/terraform-aws-eks-karpenter/main.tf line 69, in resource "kubectl_manifest" "karpenter_provisioner":
│   69: resource "kubectl_manifest" "karpenter_provisioner" {

I found in the EKS logs that a request was made to the Kubernetes API with an incorrect path, resulting in a 404 error response. It appears that the provider is making the request with an extra ApiVersion, causing it to be duplicated in the path.

  "requestURI": "/apis/karpenter.k8s.aws/v1alpha1/v1alpha1/awsnodetemplates/default",

Full log entry:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "stage": "ResponseComplete",
  "requestURI": "/apis/karpenter.k8s.aws/v1alpha1/v1alpha1/awsnodetemplates/default",
  "verb": "get",
  "user": {
    "username": "kubernetes-admin",
    "groups": [
      "system:masters",
      "system:authenticated"
    ],
    "extra": {
...
      ]
    }
  },
  "sourceIPs": [
  ],
  "userAgent": "HashiCorp/1.0 Terraform/1.5.0",
  "objectRef": {
    "resource": "v1alpha1",
    "name": "awsnodetemplates",
    "apiGroup": "karpenter.k8s.aws",
    "apiVersion": "v1alpha1",
    "subresource": "default"
  },
  "responseStatus": {
    "metadata": {},
    "code": 404
  },
  "requestReceivedTimestamp": "2023-06-14T12:07:55.982587Z",
  "stageTimestamp": "2023-06-14T12:07:55.982853Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}
+1
idontlikej on Jun 14, 2023
Read more comments on GitHub
← terraform-provider-kubectl: kubectl_manifest returning "Provider produced inconsistent result after apply" in Kubernetes 1.20
terraform-provider-kubectl: The "count" value depends on resource attributes that cannot be determined until apply →