crossplane: Experiencing degraded performance and increased load on kubernetes api server for Crossplane 1.14+

What happened?

We run Crossplane on EKS and we use AWS provider family v1.1.0. We have a very limited amount of managed resources ~ 65 in total. When we upgraded Crossplane from 1.13.2 to 1.14+ (tested with multiple releases) we observed the following:

  • Significant increase in API server load: 4x increase in read/write requests SLI, CPU usage went from 0.5 to ~2.5

    API server load screenshots Screenshot 2024-03-07 at 09 51 39 Screenshot 2024-03-07 at 09 58 33
  • Significant increase in crossplane pods network that went from 30KiB/s to spikes of 5MiB/s

    Network rate increase screenshot Screenshot 2024-03-07 at 10 26 03
  • Non negligible increase in Crossplane pods and providers resource usage:

    Crossplane + provider pods memory consumption increase screenshot Screenshot 2024-03-07 at 10 42 35

No Crossplane managed resource was being created or modified during the upgrade. After the upgrade we also started to get multiple logs of client-side throttling due to PUT request being repetitively done to https://172.20.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/[..] for every single CRD eg.:

I0307 08:19:03.050531       1 request.go:697] Waited for 1.142962507s due to client-side throttling, not priority and fairness, request: PUT:https://172.20.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/userloginprofiles.iam.aws.upbound.io

We expected no performance degradation after the upgrade.

How can we reproduce it?

The only action that caused this for us was the upgrade of Crossplane from 1.13.2 to 1.14+.

What environment did it happen in?

Crossplane version: 1.14+ EKS: 1.27

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (11 by maintainers)

Most upvoted comments

@turkenh 💡 thanks, I can see this happening. Like mentioned above, we had previously moved to DeploymentRuntimeConfigs still using the shared service account (with a shared runtime config) and we still faced the issue then. It turns out the issue was relying on a crossplane managed service account.

This was the DeploymentRuntimeConfig causing issues:

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  creationTimestamp: "2024-03-15T09:54:26Z"
  generation: 2
  name: aws-runtime-config
spec:
  deploymentTemplate:
    spec:
      selector: {}
      template:
        spec:
          containers:
          - name: package-runtime
            resources:
              limits:
                memory: 700Mi
              requests:
                cpu: 450m
                memory: 700Mi
  serviceAccountTemplate:
    metadata:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/example
      name: crossplane-generic-provider

Managing the SA externally (removing the serviceAccountTemplate), and passing the serviceAccountName to DeploymentRuntimeConfig instead fixed it.

Thank you all very much for the help here 🙇‍♀️

Ok, I think I found the problem 🕵️

You’re using a shared service account between providers, crossplane-generic-provider, and indeed hitting https://github.com/crossplane/crossplane/issues/4552 which we fixed with DeploymentRuntimeConfigs.

The gist of the problem is both providers trying to exclusively own the shared service account and adding their ownerRefs while removing the other and constantly fighting. Changes in the SA triggers reconciliation on both revisions…

So, the solution is to switch DeploymentRuntimeConfigs, https://github.com/crossplane/crossplane/issues/4908 is also related, and I believe you can use the example there.

This issue seems related to https://github.com/crossplane/crossplane/issues/5074#issuecomment-1935605535, as @turkenh was investigating an increased number of update calls on CRDs there too