controller-runtime: Server-side Apply problem

In our internal scenario, We developed a PaaS platform based on OAM our business simply declare the Workload and Traits they use There are two types of Workload, ServerWorkload(online) and TaskWorkload(offline) Traits include ManualScalerTrait,AutoScalerTrait,LoadBalanceTrait etc.

The ServerWorkload renders a Deployment and creates it When the ManualScalerTrait observes that this Deployment has been created, it changes the replicas of this Deployment

In practice, however, we found that once our Operator was restarted, the POD that was already under the running deployment would be restarted too.

found the following event

Events:
  Type    Reason             Age                  From                   Message
  ----    ------             ----                 ----                   -------
  Normal  ScalingReplicaSet  13s (x12 over 5h5m)  deployment-controller  Scaled down replica set simple-web-33-5d759bd4cf to 0
  Normal  ScalingReplicaSet  12s (x15 over 5h7m)  deployment-controller  Scaled up replica set simple-web-33-5d759bd4cf to 3

Because the replicas of Deployment rendered by ServerWorkload is 0 But the number of ManualScaler is 3, the deployment changes from 3 to 0 and then to 3 again, Caused a restart

Based on the implementation mechanism, it is difficult for ServerWorkload to patch Deployment because it renders a complete Deployment template.

If APIServer can provide this parameter client.SkipConflictFields, perfect solution

ao := []client.PatchOption{ client.FieldOwner(c1.GetUID()),client.SkipConflictFields}
err := r.Patch(ctx, deploy, client.Apply, ao...)

controller-runtime version : 0.6.2

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 17 (9 by maintainers)

Most upvoted comments

That would be a feature request for upstream, but seems unlikely. As mentioned, you shouldn’t be sending a replicas field at all. A common mistake with Apply patches is to use a normal object, you almost always want to be using an unstructued. Take a look at the code in https://github.com/coderanger/controller-utils/blob/main/components/template.go (or just use it directly)

coderanger on Dec 29, 2020

I do it in my rabbitmq-operator, https://github.com/coderanger/rabbitmq-operator/blob/main/controllers/rabbituser.go though the code is probably going to be hard to follow. Also it’s been a huge source of bugs and I really want to get rid of it 😄

coderanger on Dec 28, 2020