crossplane: Deleting a Provider before its MRs results in stuck MRs
What happened?
While working on https://github.com/crossplane/crossplane/pull/4187 I was reminded that you must ensure all of a provider’s MRs are completely deleted before uninstalling the provider. If you don’t:
- Any MRs will be left in an orphaned, unreconciled, un-deletable state.
- The orphaned MRs will prevent the CRD that defines them being removed.
- The CRD that defines them will prevent the provider being reinstalled.
I believe this happens because:
- The
Provider.pkgand its revision are deleted immediately, relying on Kubernetes garbage collection to delete the MR CRDs. - An MR CRD can’t be deleted while an MR (CR) exists. The CRD has
metadata.deletionTimestampset, but deletion is blocked by thecustomresourcecleanup.apiextensions.k8s.iofinalizer. - The
Provider.pkgand its revisions being gone means the provider deployment no longer exists. This means you now can’t delete the MR - deletion will be blocked by thefinalizer.managedresource.crossplane.iofinalizer.
I believe if you get into this state your only option is to do finalizer surgery on the MR(s) that are blocking the deletion (and to manually delete any corresponding external resources).
How can we reproduce it?
- Create a provider:
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-nop
spec:
package: xpkg.upbound.io/crossplane-contrib/provider-nop:v0.2.0
ignoreCrossplaneConstraints: true
- Create an MR:
apiVersion: nop.crossplane.io/v1alpha1
kind: NopResource
metadata:
name: example-resource
spec:
forProvider:
conditionAfter:
- conditionType: Ready
conditionStatus: "False"
conditionReason: "Creating"
time: 0s
- conditionType: Ready
conditionStatus: "True"
conditionReason: "Available"
time: 10s
- Delete the provider
- Try to delete the MR
What environment did it happen in?
Crossplane version: b65fa47e5f270b3054840cce1e0c4fcc9e96b7b7
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 4
- Comments: 15 (9 by maintainers)
I’ve hit this exact issue as well. Also found an extra layer of unexpected behaviour from Kubernetes which plays a part in making this potentially more confusing and dangerous. That is the possibility for ghost MRs, which happen in the scenario below:
customresourcecleanup.apiextensions.k8s.iofrom the CRD.finalizer.managedresource.crossplane.io.metadata.deletionTimestamp.This can obviously make this a lot more confusing and dangerous, so sharing here in case it may be of any use to anyone else.
Presumably the only way to fix this is to prevent the provider being deleted until all of its MRs are deleted. Given that MR deletion blocks MR CRD deletion, potentially we could have MR CRD deletion in turn block Provider deletion.
As to how to block deletion, I think the options are:
Option 3 seems most feasible to me for now, and I like that it’s the same pattern from Provider.pkg -> CRD -> MR. That said, I think the way MR CRD deletion works will complicate whatever we do.
I believe we currently rely on Kubernetes garbage collection to delete CRDs. This works well because CRDs are created and controlled (in the owner ref sense) by a
ProviderRevision.pkgbut also owned (again in the ref sense) by aProvider.pkg. The idea being that we don’t want a CRD to be garbage collected just because the revision that created it no longer exists:https://github.com/crossplane/crossplane/blob/b65fa47e5f270b3054840cce1e0c4fcc9e96b7b7/internal/controller/pkg/revision/establisher.go#L345
/fresh still relevant as part of the parent issue:
/fresh
/fresh
/fresh