crossplane: Experiencing degraded performance and increased load on kubernetes api server for Crossplane 1.14+
What happened?
We run Crossplane on EKS and we use AWS provider family v1.1.0. We have a very limited amount of managed resources ~ 65 in total. When we upgraded Crossplane from 1.13.2 to 1.14+ (tested with multiple releases) we observed the following:
-
Significant increase in API server load: 4x increase in read/write requests SLI, CPU usage went from 0.5 to ~2.5
API server load screenshots
-
Significant increase in crossplane pods network that went from 30KiB/s to spikes of 5MiB/s
Network rate increase screenshot
-
Non negligible increase in Crossplane pods and providers resource usage:
Crossplane + provider pods memory consumption increase screenshot
No Crossplane managed resource was being created or modified during the upgrade.
After the upgrade we also started to get multiple logs of client-side throttling due to PUT request being repetitively done to https://172.20.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/[..] for every single CRD eg.:
I0307 08:19:03.050531 1 request.go:697] Waited for 1.142962507s due to client-side throttling, not priority and fairness, request: PUT:https://172.20.0.1:443/apis/apiextensions.k8s.io/v1/customresourcedefinitions/userloginprofiles.iam.aws.upbound.io
We expected no performance degradation after the upgrade.
How can we reproduce it?
The only action that caused this for us was the upgrade of Crossplane from 1.13.2 to 1.14+.
What environment did it happen in?
Crossplane version: 1.14+ EKS: 1.27
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 17 (11 by maintainers)
@turkenh 💡 thanks, I can see this happening. Like mentioned above, we had previously moved to
DeploymentRuntimeConfigsstill using the shared service account (with a shared runtime config) and we still faced the issue then. It turns out the issue was relying on a crossplane managed service account.This was the
DeploymentRuntimeConfigcausing issues:Managing the SA externally (removing the
serviceAccountTemplate), and passing the serviceAccountName toDeploymentRuntimeConfiginstead fixed it.Thank you all very much for the help here 🙇♀️
Ok, I think I found the problem 🕵️
You’re using a shared service account between providers,
crossplane-generic-provider, and indeed hitting https://github.com/crossplane/crossplane/issues/4552 which we fixed withDeploymentRuntimeConfigs.The gist of the problem is both providers trying to exclusively own the shared service account and adding their ownerRefs while removing the other and constantly fighting. Changes in the SA triggers reconciliation on both revisions…
So, the solution is to switch
DeploymentRuntimeConfigs, https://github.com/crossplane/crossplane/issues/4908 is also related, and I believe you can use the example there.This issue seems related to https://github.com/crossplane/crossplane/issues/5074#issuecomment-1935605535, as @turkenh was investigating an increased number of update calls on CRDs there too