operator: Tigera Operator unable to uninstall cleanly

installations.operator.tigera.io/default blocks uninstallation in my case.

kubectl patch -n calico-system installations.operator.tigera.io/default  --type json \
  --patch='[{"op":"remove","path":"/metadata/finalizers"}]'

Add some my experiences for uninstalling calico tigera-operator in our EKS cluster.

1. run helm uninstall to uninstall tigera-operator but only tigera-operator get uninstalled and other resource in namespace calico-system still exist. (same issue with this thread)
2. run helm install again to install tiger-operator, then all the resources in namespace calico-system get cleaned, but only tigera-operator is there in the cluster.
3. run helm uninstall to uninstall tigera-operator again, and then everything is gone.

That’s what I did for uninstalling calico as a workaround and I haven’t dug into a bit for the detailed behaviors of those yet. Not sure if anybody else doing this before.

After uninstalling the operator with Helm and cleaning up the rest with kubectl, I’m stuck with a calico-node ServiceAccount in the calico-system namespace that refuses to be deleted. How can I delete it?

Edit: Solved. I had to patch away the finalizer.

kubectl patch -n calico-system ServiceAccount/calico-node --type json \
  --patch='[{"op":"remove","path":"/metadata/finalizers"}]'

+1

For us, installations.operator.tigera.io/default finalizers block tigera helm uninstallation and ServiceAccount/calico-node finalizers block deletion of the calico_system namespace

Deleting the installations.operator.tigera.io default before destroying the helm tigera, does remove installations and also removes the finalizers on calico-system ServiceAccount/calico-node and destroys cleanly.

resource "helm_release" "tigera_calico" {
  name       = "tigera-calico-release"
  chart      = "tigera-operator"
  repository = "https://projectcalico.docs.tigera.io/charts"
  namespace  = "tigera-operator"
  timeout    = 300
  version    = "3.25.0"

  depends_on = [
    kubernetes_namespace.tigera_operator,
    kubernetes_namespace.calico_system,
    kubernetes_namespace.calico_apiserver
    ]
  set {
    name  = "installation.kubernetesProvider"
    value = "EKS"
  }
}

resource "null_resource" "remove_finalizers" {
  depends_on = [helm_release.tigera_calico]

  provisioner "local-exec" {
    when    = destroy
    command = <<-EOT
      kubectl delete installations.operator.tigera.io default
    EOT
  }

  triggers = {
    helm_tigera = helm_release.tigera_calico.status
  }
}

@Rambatino in this case it’s explicitly a stuck resource with no associated controller to manage it. I agree that removing finalizers is generally a bad idea but in this case, given the cluster is likely in the process of being destroyed, it’s the lesser of two evils. That said if you’re not destroying the cluster and are removing the finalizer to delete the installation you’re going to need to either replace all the nodes or manually fix them.

Example PR with one approach for resolving this: https://github.com/tigera/operator/pull/2662/

That doc doesn’t seem to say explicitly that you shouldn’t use Method 1, but does provide some caveats / tradeoffs.

It doesn’t (anymore), but although it works for day 0 (kind of) it doesn’t allow for CRD updates so is functionally useless for a project like Tigera Operator where the CRDs change.

Hm, yeah that could be a problem if helm doesn’t order the deletes. Is helm not smart enough to wait for termination of the deployment before deleting the resources that the deployment depends on?

Helm understands built in resource types and ordering so should be good on that front. I also think you can’t delete a resource which is being used by a pod.

If neither of these helps then maybe the operator can have a termination wait arg added so it can process any CRs deleted at the same time before terminating.

This would have been my first thought as well - add a termination grace period, have the operator handle SIGTERM and rather than exit immediately, delay exiting until it is confident it doesn’t need to remove any more finalizers or do any further cleanup. In normal operation that should happen pretty quickly. If we hit the end of the grace period (maybe 60s or so) then we’ll get a SIGKILL and be forced to shutdown.

@SamuZad although you’re technically correct (well assuming enough of a pause between the kubectl delete and helm uninstall); the point here is that installation method can’t also support uninstallation which isn’t great UX and means it isn’t suitable for declarative IaC.

The chart stevehipwell/tigera-operator is not supported by the tigera/operator team. But I expect that this would probably be an issue with the official helm chart also. I’m guessing this is because the operator deployment is being removed, so the finalizer it adds, is not being removed. I think first removing the operator CustomResource (CR) (the Installation “default” resource) and allowing the operator to remove the finalizer then removing the operator I believe would work.

@caseydavenport do you know what the correct way to handle ensuring the operator deployment isn’t removed before the Installation CR is since it puts a finalizer on the CR? Should the operator be putting a finalizer on itself (the operator deployment) also? That seems like a bad idea to me, if that was the right way then it would probably need to put a finalizer on all the resources it uses to prevent helm from removing them before it could clean up too.

Maybe there is a helm chart feature that will delete the Installation CR and ensure it is removed before deleting everything else? (I am doubtful of this since I don’t think helm has much in the way of being able to sequence things.)

Maybe we just need to remove the use of the finalizer which I added in #1710, though from what was trying to be fixed it seems like the same issue would be hit in this use case too.

What is the correct order to uninstall the complete manifest? I’ve deployed via kubectl

You should uninstall custom-resources.yaml first, and then tigera-operator.yaml - just the opposite of install ordering.

If you uninstall the operator first, then it won’t be running in the cluster in order to clean up after itself when custom-resources.yaml is deleted.