crossplane: ProviderConfigs should not be deleted until all resources referring to it are deleted

I believe this is a close call complexity wise, and is subjective from a UX perspective - especially if we can agree that it’s worth making the consumers of a ProviderConfig easily discoverable even when it is not being deleted.

If we go with the ProviderConfigUsage approach:

• In the happy path - the vast majority of cases - we make one API call to create a ProviderConfigUsage when a managed resource first connects. Subsequent connects may read from cache and avoid a no-op update.
• The ProviderConfig controller can determine whether it should remove its finalizer and inform users whether it’s in use by listing one type - ProviderConfigUsage - from cache. The list can be filtered by a well known label.
• The ProviderConfig controller must watch two types - ProviderConfig and ProviderConfigUsage - in order to react immediately when a ProviderConfig may be safely deleted.

If we invert the responsibility of discovering usages such that the ProviderConfig controller handles it:

• The ProviderConfig controller must be aware of the N types that may use a ProviderConfig. We can probably register these types when instantiating the controller to avoid having to use API discovery or similar.
• The ProviderConfig controller must make N list calls from cache to list N heterogeneous types, and parse the spec.providerConfigRef from each resource of each type to determine whether it uses the provider.
• The ProviderConfig controller must watch 1+N types - ProviderConfig and the N types that may use it - in order to react immediately when a ProviderConfig may be safely deleted.

So complexity wise it seems like the ProviderConfigUsage approach trades increased storage (many instances of a small resource) for reduced compute (fewer types to list and parse, fewer watches).

UX wise it gets pretty subjective. The ProviderConfigUsage approach is arguably the more idiomatic - it reflects the canonical deployment-to-pod relationship in which the user determines which pods are part of a deployment by running kubectl get pod -l some=labelselector, not by looking at an array of pod references in the Deployment’s status. There are also parallels with the RBAC RoleBinding type that binds a role to one or more subjects.

If the ProviderConfig controller were responsible for discovering which resources were using it they would presumably be displayed as an array of object references in the status of a ProviderConfig. This the more common approach in Crossplane controllers (e.g. the relationship between a composite resource and its composed resources, or a package revision and the resources it installs). One variation here is that the composite resource and package revision Crossplane controllers are responsible for the lifecycle of the resources they track via object references - they create them, rather than discover them after the fact.

My assessment is that neither approach is objectively better from a performance standpoint. I still feel that the ProviderConfigUsage approach is slightly better from a UX and implementation-complexity standpoint.

Ah - I like the potential for symmetry with storage protection. It would be good for providers to similarly transition to a specific status to indicate they’re being removed, perhaps Status: Terminating would work for them too.