cluster-api: Slow cluster creation when working with hundreds of clusters

Thanks! I’ll close this then. It will be easier to track specific issues that way. This has been a great discussion and exploration, thanks!

@lentzi90 those are great insights!

And we definitely need to join forces given that this work is relevant for the entire community cc @sbueringer @killianmuldoon

If you have any ideas for how to lower the memory usage of these API servers, or maybe other ways to fake them, I would love to hear about it!

I don’t think we can fit as much as we want in a single machine, and continuing down this path also introduces some other issues like the noise from the fake workload clusters which can affect the management cluster and test results.

What we are doing in the kubemark provider is introducing the idea of a “backing clusters”, which are one or more external clusters which are providing the necessary computing power to run all the fake workloads clusters (also K8s scale tests are using a simular approach).

By moving fake clusters to external clusters you can potentially scale indefinitely, and you can always fallback on testing everything on one machine when you are working with less power angry test targets or for smoke tests.

Then to conserve resources I set up a multi-tenant etcd to back all the API servers. (Learning a lot here! 😅 )

That’s a great idea, we should definitely embed this on https://github.com/kubernetes-sigs/cluster-api-provider-kubemark/issues/63 as soon as I get to implementing the control plane part, if you don’t get there before me 😜

/triage accepted Great work of research! The next step is to translate this into actionable improvements on KCP or other controllers. Metrics and the work on logs will help in doing so, but this is a awesome start

For me it is ok to close it but if you want to keep it for more discussions that is also fine 🙂 I know there is work on-going with adding scalability e2e tests, maybe that can be tracked here? I marked the PR as fixing this since it solved the main blocker for my use case, but there is definitely more to do

This is extremely useful/helpful to me, thank you for the hard work!

cc @richardcase who might be interested in the discussion as well

Thanks @fabriziopandini , very timely. This is gold @lentzi90 , great work and super helpful.

Might be worth looking if the work queue length metrics shows supicious behavior (e.g. goes up non-linear at some point)

Those are again great insights, @lentzi90 thanks for sharing. I agree that kcp cpu and memory consumption is something to be investigated. I’m not sure about the correlation between the number of clusters and qps, because my assumption is that a cluster will “stop” being reconciled as soon as it is provisioned, so it should not clog the reconcile queue (with the exception of resync event every 10 minutes). But this is when the work on metrics becomes relevant for finding bottlenecks and also to explaining why provisioning time is degrading.

If this is ok for you It would be great to set up some time to discuss the possible next step of this work, and possibly how to upstream it. We can also discuss this at KubeCon if you are planning to make it, otherwise, I will be happy to set up something using the CAPI project zoom, so we can also record and share with the others members of the community.

Thanks for the comment @sbueringer !

My first guess would be that increasing --kubeadmcontrolplane-concurrency should improve the situation.

This is a good point. I should probably have tried that before sharding… I will try it now and see if it helps. 🙂

I assume with appear you mean that the Machine objects are not even created at this point?

Exactly! Here I managed to capture what it looks like. When the KubeadmControlPlane is 63 seconds old, there is no Machine and the SA and proxy secrets have just been created. After this the Machine (and Metal3Machine) appears.

$ kubectl -n test-50 get kcp,machine,m3m,secret,m3d
NAME                                                        CLUSTER   INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/test-50   test-50                                                                                   63s   v1.25.3

NAME                                   TYPE                      DATA   AGE
secret/test-50-apiserver-etcd-client   kubernetes.io/tls         2      65s
secret/test-50-ca                      kubernetes.io/tls         2      66s
secret/test-50-etcd                    kubernetes.io/tls         2      66s
secret/test-50-proxy                   cluster.x-k8s.io/secret   2      1s
secret/test-50-sa                      cluster.x-k8s.io/secret   2      1s
secret/worker-1-bmc-secret             Opaque                    2      66s
$ kubectl -n test-50 get kcp,machine,m3m,secret,m3d
NAME                                                        CLUSTER   INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE   VERSION
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/test-50   test-50                                                                                   67s   v1.25.3

NAME                                     CLUSTER   NODENAME   PROVIDERID   PHASE   AGE   VERSION
machine.cluster.x-k8s.io/test-50-fg498   test-50                                   0s    v1.25.3

NAME                                                                       AGE   PROVIDERID   READY   CLUSTER   PHASE
metal3machine.infrastructure.cluster.x-k8s.io/test-50-controlplane-x6rnj   0s                         test-50

NAME                                   TYPE                      DATA   AGE
secret/test-50-apiserver-etcd-client   kubernetes.io/tls         2      69s
secret/test-50-ca                      kubernetes.io/tls         2      70s
secret/test-50-etcd                    kubernetes.io/tls         2      70s
secret/test-50-kubeconfig              cluster.x-k8s.io/secret   1      3s
secret/test-50-proxy                   cluster.x-k8s.io/secret   2      5s
secret/test-50-sa                      cluster.x-k8s.io/secret   2      5s
secret/worker-1-bmc-secret             Opaque                    2      70s

First of all, great work. Nice to see that folks are starting to test CAPI at scale! 😃

There is a long pause after the KCP is created before the Machines appear.

I assume with appear you mean that the Machine objects are not even created at this point?

It looks to me like you are running Cluster API in the default configuration. My first guess would be that increasing --kubeadmcontrolplane-concurrency should improve the situation. The default is 10, which means KCP can only reconcile 10 KCP objects at the same time. All others have to wait.

The next question would be what the 10 workers are actually doing. It might be that they are blocked in some way.

Thank you @killianmuldoon ! I’m following https://github.com/kubernetes-sigs/cluster-api-provider-kubemark/issues/63 with great interest! Should have thought to include it in the issue directly… Let me know if you have time to try it, and if there are any issues!

Most of these tests have been on a cloud VM with 32 GB memory and 8 core CPU. It didn’t look like resource contention (except when I did the sharding, that maxed out the CPU).

This is amazing work! There’s been a lot of questions around scaling recently so this is really useful, and definitely the best attempt so far at reproducible scale tests. I’m excited to see if I can get this running when I get time.

It would be interesting to be able to profile the CAPI controller code while this is running. BTW - what size of a machine was this running on, and was it resource contention or just slowness in the controller that caused the scaling issue?

Here’s a couple of related issues for reference. https://github.com/kubernetes-sigs/cluster-api/issues/7308 https://github.com/kubernetes-sigs/cluster-api-provider-kubemark/issues/63