spinnaker: Clouddriver + kube v2 provider: cannot delete unregistered CRDs

Issue Summary:

Unregistered CRDs cannot be deleted using the Delete (Manifest) stage

Cloud Provider(s):

Kubernetes V2

Environment:

Spinnaker 1.12.8 Spinnaker 1.13.1

Feature Area:

Clouddriver

Steps to Reproduce:

Attempt to use the Delete (Manifest) stage targeting a Unregistered CRD such as ServiceMonitor or VerticalPodAutoscaler

Sample Stage

{
  "account": "my-account",
  "app": "hello-world",
  "cloudProvider": "kubernetes",
  "completeOtherBranchesThenFail": false,
  "continuePipeline": false,
  "failPipeline": false,
  "kinds": [
    "ingress",
    "ServiceMonitor",
    "horizontalPodAutoscaler",
    "VerticalPodAutoscaler",
    "podDisruptionBudget",
    "service"
  ],
  "labelSelectors": {
    "selectors": [
      {
        "key": "app",
        "kind": "EQUALS",
        "values": [
          "${trigger.parameters['app']}"
        ]
      },
      {
        "key": "release",
        "kind": "EQUALS",
        "values": [
          "${trigger.parameters['release']}"
        ]
      }
    ]
  },
  "location": "${trigger.parameters['artifactLocation']}",
  "mode": "label",
  "name": "Delete Other Resources",
  "options": {
    "cascading": true
  },
  "type": "deleteManifest"
}

In Spinnaker 1.12.8

"exception": {
"cause": "KubectlException",
"message": "Failed to delete app = web-application,release = helloworld from apps: error: the server doesn't have a resource type \"none\"\n",
"operation": "KubernetesDeleteManifestOperation",
"type": "EXCEPTION"
}
2019-04-03 15:40:41.208  INFO 1 --- [ol-2-thread-284] c.n.s.c.data.task.jedis.JedisTask        : [ORCHESTRATION] Orchestration failed: KubernetesDeleteManifestOperation | KubectlException: [Failed to delete app = web-application,release = helloworld from apps: error: the server doesn't have a resource type "none"]
2019-04-03 15:40:41.209 ERROR 1 --- [ol-2-thread-284] c.n.s.c.o.DefaultOrchestrationProcessor  : com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor$KubectlException: Failed to delete app = web-application,release = helloworld from apps: error: the server doesn't have a resource type "none"
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.job.KubectlJobExecutor.delete(KubectlJobExecutor.java:180)
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.lambda$delete$14(KubernetesV2Credentials.java:526)
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.runAndRecordMetrics(KubernetesV2Credentials.java:571)
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.runAndRecordMetrics(KubernetesV2Credentials.java:562)
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.security.KubernetesV2Credentials.delete(KubernetesV2Credentials.java:526)
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.handler.CanDelete.delete(CanDelete.java:38)

In Spinnaker 1.13.1

"exception": {
"cause": "NullPointerException",
"message": "java.lang.NullPointerException\n\tat com.netflix.spinnaker.clouddriver.kubernetes.v2.op.manifest.KubernetesDeleteManifestOperation.lambda$operate$0(KubernetesDeleteManifestOperation.java:68)\n\tat java.util.ArrayList.forEach(ArrayList.java:1257)\n\tat com.netflix.spinnaker.clouddriver.kubernetes.v2.op.manifest.KubernetesDeleteManifestOperation.operate(KubernetesDeleteManifestOperation.java:65)\n\tat com.netflix.spinnaker.clouddriver.kubernetes.v2.op.manifest.KubernetesDeleteManifestOperation.operate(KubernetesDeleteManifestOperation.java:35)\n\tat com.netflix.spinnaker.clouddriver.orchestration.AtomicOperation$operate.call(Unknown Source)\n\tat org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)\n\tat com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy:89)\n\tat com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1$_closure2.doCall(DefaultOrchestrationProcessor.groovy)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)\n\tat groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)\n\tat org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)\n\tat groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)\n\tat groovy.lang.Closure.call(Closure.java:418)\n\tat groovy.lang.Closure.call(Closure.java:412)\n\tat com.netflix.spinnaker.clouddriver.metrics.TimedCallable$ClosureWrapper.call(TimedCallable.groovy:55)\n\tat com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.groovy:82)\n\tat java_util_concurrent_Callable$call.call(Unknown Source)\n\tat org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)\n\tat com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy:88)\n\tat com.netflix.spinnaker.clouddriver.orchestration.DefaultOrchestrationProcessor$_process_closure1.doCall(DefaultOrchestrationProcessor.groovy)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)\n\tat groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)\n\tat org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)\n\tat groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)\n\tat groovy.lang.Closure.call(Closure.java:418)\n\tat groovy.lang.Closure.call(Closure.java:412)\n\tat com.netflix.spinnaker.security.AuthenticatedRequest.lambda$propagate$0(AuthenticatedRequest.java:97)\n\tat com.netflix.spinnaker.clouddriver.metrics.TimedCallable.call(TimedCallable.groovy:82)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n",
"operation": "KubernetesDeleteManifestOperation",
"type": "EXCEPTION"
},
2019-04-03 15:43:55.910 ERROR 1 --- [pool-2-thread-2] c.n.s.c.o.DefaultOrchestrationProcessor  : java.lang.NullPointerException
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.manifest.KubernetesDeleteManifestOperation.lambda$operate$0(KubernetesDeleteManifestOperation.java:68)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.manifest.KubernetesDeleteManifestOperation.operate(KubernetesDeleteManifestOperation.java:65)
	at com.netflix.spinnaker.clouddriver.kubernetes.v2.op.manifest.KubernetesDeleteManifestOperation.operate(KubernetesDeleteManifestOperation.java:35)

Additional Details:

The Unregistered CRD’s will Deploy (Manifest) just fine, however they cannot be cleaned up.

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (5 by maintainers)

Most upvoted comments

I finally got this to work by Specifying both kinds and customResources in my kubernetes account definition.

I had to re-declare all of the default kinds listed in https://github.com/spinnaker/clouddriver/blob/master/clouddriver-kubernetes/src/main/groovy/com/netflix/spinnaker/clouddriver/kubernetes/v2/description/manifest/KubernetesKind.java otherwise I was blocked from deploying/deleting those

        kinds:
          - ApiService
          - ClusterRole
          - ClusterRoleBinding
          - ConfigMap
          - ControllerRevision
          - CustomResourceDefinition
          - CronJob
          - DaemonSet
          - Deployment
          - Event
          - HorizontalPodAutoscaler
          - Ingress
          - Job
          - MutatingWebhookConfiguration
          - Namespace
          - NetworkPolicy
          - PersistentVolume
          - PersistentVolumeClaim
          - Pod
          - PodPreset
          - PodSecurityPolicy
          - PodDisruptionBudget
          - ReplicaSet
          - Role
          - RoleBinding
          - Secret
          - Service
          - ServiceAccount
          - StatefulSet
          - StorageClass
          - ValidatingWebhookConfiguration
          - ServiceMonitor.monitoring.coreos.com
          - VerticalPodAutoscaler.autoscaling.k8s.io
        omitKinds: []
        customResources:
          - kubernetesKind: ServiceMonitor.monitoring.coreos.com
            versioned: false
          - kubernetesKind: VerticalPodAutoscaler.autoscaling.k8s.io
            versioned: false