crossplane: Proposal: Break up large providers by service

Naively and from user experience perspective, I would say having a single provider with the ability to control CRDs - and so the resources to watch in the controller - via an allow list - default to * to keep backward compatibility - feels the most ideal. Controlling even to the specific CRD level would be nice like: buckets.s3.aws.upbound.io or to get a whole group say *.s3.aws.upbound.io. Managing providers from e.g. Flux it would be nice this way to simply extend / control what we use in given clusters by Crossplane adding the new CRDs and the new provider deployment starts watching them. One drawback here I could think of is removing a CRD / group from the allow list that actually cannot be removed, because let’s say it is being used and then should the provider deployment stop managing that? Might not be very difficult to - tho possibly resource intensive - validate the allow list from an admission controller.

Introducing a bunch of version / package management could possibly lead to more overhead, incompatibility and hard to debug issues. A lot of new deployments / pods for sub-providers means harder to track / correlate what is going on based on the logs.

It would be nice to be able to maintain the ProviderConfigs at the cloud-provider level instead of the API grouping level. We have to maintain many different ProviderConfigs for different accounts, and duplicating those across the API-group-providers would not be pleasant. Can the API-group-providers be “children” of the larger “container” provider and inherit the ProviderConfigs associated with it?

The additional provider processes don’t concern me as they will all use (small) fractions of the resources that the larger providers do now. If we manage the same number of cloud resources across a dozen API group providers there will be some additional overhead but not enough to be concerned about.

Ideally we could use wildcards for versioning / groups, so that this is much less of a burden. Something like:

includedAPIs:
- group: "*.rds.upbound.io"
  apiVersions:
  - v1
  - "*beta*"
  - v2alpha1
- group: "*.ec2.upbound.io"
  apiVersions:
  - "*" 

Consider a company that has gone all-in on using crossplane for managing infrastructure. The micro-controller based approach would lead to a very large system footprint if that company were to use multiple cloud (and …other…) crossplane providers.

I agree that from a user perspective it would be nicer to just deal with one provider-aws (that is one ProviderConfig to install and ideally stay with one pod per provider, as there will be more idling resources ‘wasted’ if we run one pod per cloud service).

But that made me think: can’t we make the on/off switches for services via changes in the already existing APIs? I’m thinking of the Provider resource. When you install a provider (even via CLI I believe) this resource is created and the crossplane core controller decides which CRDs to install and which Deployment (further configured by the ControllerConfig) to create.

So in the Provider resource there could be a new service field that allows to filter the services that should be installed. This field should be optional to not influence providers that don’t need service filtering (like the helm and terraform providers) and also to maintain backwards compatibility.

This service field could be used by the crossplane core controller to only install a subset of the CRDs in the package. It would also need to pass this information onto the actual provider Deployment, as the actual provider binary would need to be configured to watch only a subset of service resources. This could be done either via env vars or binary flags passed onto the Deployment.

I think overall this approach would cover all the positive aspects that you mentioned initially, but also wouldn’t let the amount of providers explode. The marked place would stay more lean (still one provider per cloud) and services could be switched on and off more easily by the user (just add/remove to/from the list rather than installing a new ‘provider’). This is probably besides the point, but the current terminology ‘provider’ just fits better describing the whole cloud provider. But I think it makes total sense to have a separation by service inside that resource.

It feels like increase in the cognitive complexity is unavoidable but most of them can be addressed with better tooling or more changes. For example, I’m not too worried about declared dependencies of a Configuration package since we can expand that dependency declaration to include certain CRDs if we wanted to go with filtering. Similarly, proliferation of ProviderConfig type in case of separate package option is reasonably avoidable with a common package dependency.

However, the following increases in cognitive complexity seems unavoidable:

• Filtering causes you to think more when you install the provider as well as every time you use a new service since you need to check whether it’s installed. Over time, we’ll need to bring more focus to availability of CRDs rather than providers in a cluster, which is a mental shift that has trickling effects - dependencies of Configuration being one of them.
• Package-per-service brings in version compatibility problems to the table. There are clever ways to make this easier but it’s a new problem and a hard one - there are bi-directional dependencies and possibilities of a deadlock in cases where you manage many different versions.
  • https://github.com/crossplane/crossplane/issues/1770 as suggested by @negz removes the cross-resource dependencies in the code but inherently there is always API dependency, i.e. if the field is changed or moved, you need to change the field path in your reference (in fact, if API isn’t changed, the frequency of break of compatibility is same if we kept cross-resource references as-is since the code that resolves the references can unmarshal the JSON as long as the field is there). Today, all this is handled behind the scenes since the field path is not exposed.
  • Worth noting that, from performance perspective, I’d prefer multiple pods to increase the mobility of the workload - there are cases where provider needs 8GB or more RAM and that starts to impose requirements in regards to what type of instances you can use.

I don’t love the filtering option but feels like we have to pay some cost and the package-per-service option seems to bring more cognitive complexity especially in Day 2 operations compared to filtering - I’d like to avoid having a new version compatibility problem as much as possible.

(FWIW, there was a draft implementation of filtering a while back https://github.com/crossplane/crossplane/pull/2646 )

Are we assuming that if we supported filtering we would not allow folks to install two versions of a provider at once? (e.g. provider-aws:v1.0.0 filtered to only RDS instances, and provider-aws:2.0.0 filtered to only EC2 networking stuff?)

I was thinking through this today and realized this probably wouldn’t be possible. I believe the package manager wouldn’t let two providers share the same ProviderConfig CRD.

I don’t have an exact count yet but I’m estimating ~30 unique, if grouped by family (such as *kafka.aws.upbound.io or rds.aws.upbound.io) more like 15-20.

Greatly prefer the route of filtering installed CRDs along with enabling reconcilers within a single binary.

The ACK project went the route of 1:1 AWS-Service-to-Kubernetes-Controller-Process and IMO it has bloated the resource utilization of the system as a whole. Controller-based infrastructure systems already suffer from greater overhead when compared to point-in-time systems like terraform. This would further that divide.

IMO, following the model of the Kubernetes controller-manager with its bundled reconcilers leads a simpler and more efficient system.