crossplane: ProviderConfigUsage does not work when deletion policy is set to Foreground

What happened?

While testing my function tee prototype with a claim using compositeDeletePolicy: Foreground, I noticed that ProviderConfigUsage does not work as expected.

So, my function outputs a provider kubernetes ProviderConfig resource (with InjectedIdentity) and an Object wrapping a ConfigMap and using that ProviderConfig. When I delete my claim, I noticed that the Object resource is orphaned since the ProviderConfig was deleted before the Object itself. I verified that the ProviderConfig resource had the in-use.crossplane.io finalizer and the corresponding ProviderConfigUsage resource existed.

After some investigation and thought, I believe this is a bug with the current implementation of ProviderConfigUsage when it is used with deletion policy Foreground. The issue is, with this mode of deletion propagation, resources are being deleted starting from the bottom of the tree and hence ProviderConfigUsage being owned by the Object is deleted before the actual Object is deleted. This removes the blockage on ProviderConfig and it got deleted without waiting the deletion of the Object.

@bobh66 I am wondering if you had encountered something like this before?

How can we reproduce it?

kubectl apply -f setup.yaml
kubectl apply -f claim.yaml
# wait until claim is ready
kubectl delete -f claim.yaml

setup.yaml
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-nop
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-nop:v0.2.0
  ignoreCrossplaneConstraints: true
---
apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-dummy
spec:
  # NOTE(negz): This is currently manually pushed. See README.md at
  # https://github.com/crossplane-contrib/function-dummy.
  package: xpkg.upbound.io/crossplane-contrib/function-dummy:v0.2.0
---
apiVersion: pkg.crossplane.io/v1beta1
kind: Function
metadata:
  name: function-tee
spec:
  package: turkenh/function-tee:v0.1.0
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xnopresources.nop.example.org
spec:
  group: nop.example.org
  names:
    kind: XNopResource
    plural: xnopresources
  claimNames:
    kind: NopResource
    plural: nopresources
  connectionSecretKeys:
  - test
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
     openAPIV3Schema:
       type: object
       properties:
        spec:
          type: object
          properties:
            coolField:
              type: string
          required:
          - coolField
        status:
          type: object
          properties:
            coolerField:
              type: string
---
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: xnopresources.nop.example.org
spec:
  compositeTypeRef:
    apiVersion: nop.example.org/v1alpha1
    kind: XNopResource
  mode: Pipeline
  pipeline:
  - step: be-a-dummy
    functionRef:
      name: function-dummy
    input:
      apiVersion: dummy.fn.crossplane.io/v1beta1
      kind: Response
      # This is a YAML-serialized RunFunctionResponse. function-dummy will
      # overlay the desired state on any that was passed into it.
      response:
        desired:
          composite:
            resource:
              status:
                coolerField: "I'M COOLER!"
          resources:
            nop-resource-1:
              resource:
                apiVersion: nop.crossplane.io/v1alpha1
                kind: NopResource
                spec:
                  forProvider:
                    conditionAfter:
                    - conditionType: Ready
                      conditionStatus: "False"
                      time: 0s
                    - conditionType: Ready
                      conditionStatus: "True"
                      time: 10s
              ready: READY_TRUE
        results:
         - severity: SEVERITY_NORMAL
           message: "I am doing a compose!"
  - step: tee
    functionRef:
      name: function-tee
claim.yaml
apiVersion: nop.example.org/v1alpha1
kind: NopResource
metadata:
  namespace: default
  name: apiextensions-composition-functions
spec:
  coolField: "I'm cool!"
  # This is necessary to ensure the claim's MRs are actually gone before we
  # delete the Provider - https://github.com/crossplane/crossplane/issues/4251
  compositeDeletePolicy: Foreground

What environment did it happen in?

Crossplane version: v1.14.0-rc.0.369.g293439e0

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 8
  • Comments: 22 (14 by maintainers)

Most upvoted comments

Since the ProviderConfigUsage is owned by the Managed Resource, and does not have a finalizer, it is deleted immediately when the MR is marked deleted. This opens up a window where the MR exists but the ProviderConfig has no knowledge of it so the ProviderConfig could be deleted while the MR is still trying to delete the external resource.

I think it would be better to put a finalizer on the ProviderConfigUsage and have crossplane-runtime remove the finalizer after a successfull call to the Delete method.

/fresh

/fresh

/fresh

/fresh