crossplane: Deleting a Provider before its MRs results in stuck MRs

What happened?

While working on https://github.com/crossplane/crossplane/pull/4187 I was reminded that you must ensure all of a provider’s MRs are completely deleted before uninstalling the provider. If you don’t:

  • Any MRs will be left in an orphaned, unreconciled, un-deletable state.
  • The orphaned MRs will prevent the CRD that defines them being removed.
  • The CRD that defines them will prevent the provider being reinstalled.

I believe this happens because:

  • The Provider.pkg and its revision are deleted immediately, relying on Kubernetes garbage collection to delete the MR CRDs.
  • An MR CRD can’t be deleted while an MR (CR) exists. The CRD has metadata.deletionTimestamp set, but deletion is blocked by the customresourcecleanup.apiextensions.k8s.io finalizer.
  • The Provider.pkg and its revisions being gone means the provider deployment no longer exists. This means you now can’t delete the MR - deletion will be blocked by the finalizer.managedresource.crossplane.io finalizer.

I believe if you get into this state your only option is to do finalizer surgery on the MR(s) that are blocking the deletion (and to manually delete any corresponding external resources).

How can we reproduce it?

  1. Create a provider:
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-nop
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-nop:v0.2.0
  ignoreCrossplaneConstraints: true
  1. Create an MR:
apiVersion: nop.crossplane.io/v1alpha1
kind: NopResource
metadata:
  name: example-resource
spec:
 forProvider:
   conditionAfter:
   - conditionType: Ready
     conditionStatus: "False"
     conditionReason: "Creating"
     time: 0s
   - conditionType: Ready
     conditionStatus: "True"
     conditionReason: "Available"
     time: 10s
  1. Delete the provider
  2. Try to delete the MR

What environment did it happen in?

Crossplane version: b65fa47e5f270b3054840cce1e0c4fcc9e96b7b7

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 15 (9 by maintainers)

Commits related to this issue

Most upvoted comments

I’ve hit this exact issue as well. Also found an extra layer of unexpected behaviour from Kubernetes which plays a part in making this potentially more confusing and dangerous. That is the possibility for ghost MRs, which happen in the scenario below:

  1. Starting from the original scenario in this issue, where you have MRs stuck in deletion.
  2. One way to get rid of the CRDs and all their child MRs is to remove the finalizer customresourcecleanup.apiextensions.k8s.io from the CRD.
  3. Once the CRD has no finalizer, Kubernetes happily deletes the CRD itself. However the CRs (MRs) for that CRD aren’t deleted as they are still blocked on the finalizer finalizer.managedresource.crossplane.io.
  4. While the MRs weren’t deleted, they aren’t queriable anymore because there’s no CRD to provide a schema for them any longer. If you do recreate the CRD though, the “ghost” MRs reappear, still marked for deletion with the original metadata.deletionTimestamp.

This can obviously make this a lot more confusing and dangerous, so sharing here in case it may be of any use to anyone else.

Presumably the only way to fix this is to prevent the provider being deleted until all of its MRs are deleted. Given that MR deletion blocks MR CRD deletion, potentially we could have MR CRD deletion in turn block Provider deletion.

As to how to block deletion, I think the options are:

  1. Wait for https://github.com/kubernetes/enhancements/pull/2840
  2. Some variant of https://github.com/crossplane/crossplane/pull/4215?
  3. Have the Provider.pkg controller refuse to remove its finalizer while it still owns/controls CRDs.

Option 3 seems most feasible to me for now, and I like that it’s the same pattern from Provider.pkg -> CRD -> MR. That said, I think the way MR CRD deletion works will complicate whatever we do.

I believe we currently rely on Kubernetes garbage collection to delete CRDs. This works well because CRDs are created and controlled (in the owner ref sense) by a ProviderRevision.pkg but also owned (again in the ref sense) by a Provider.pkg. The idea being that we don’t want a CRD to be garbage collected just because the revision that created it no longer exists:

https://github.com/crossplane/crossplane/blob/b65fa47e5f270b3054840cce1e0c4fcc9e96b7b7/internal/controller/pkg/revision/establisher.go#L345

$ k get crds nopresources.nop.crossplane.io -o json| jq .metadata.ownerReferences
[
  {
    "apiVersion": "pkg.crossplane.io/v1",
    "blockOwnerDeletion": true,
    "controller": true,
    "kind": "ProviderRevision",
    "name": "provider-nop-552a394a8acc",
    "uid": "5b5a955e-609d-498c-b388-d0f779aa28d8"
  },
  {
    "apiVersion": "pkg.crossplane.io/v1",
    "blockOwnerDeletion": true,
    "controller": false,
    "kind": "Provider",
    "name": "provider-nop",
    "uid": "ddaf49c9-f618-48a1-b7ff-bdb4f69291fd"
  }
]

/fresh still relevant as part of the parent issue:

/fresh

/fresh

/fresh