kubernetes: Fluentd-scaler causing fluentd pod deletions and messes with ds-controller

Forking from https://github.com/kubernetes/kubernetes/issues/60500#issuecomment-373121164:

To summarize, here’s what we observed:

1. PATCH daemonset calls coming every minute from both fluentd-scaler and addon-manager (verified this by individually turning them on/off). Things we need to understand here:

• Is the daemonset object continuously toggling b/w 2 states? (we know here that it’s RV is increasing continuously)
• If yes, what field(s) in the object are changing? IIRC value of some label/annotation (I think ‘UpdatedPodsScheduled’) is changing (probably related to 2 below)
• Also, why/should the fluentd-scaler send any api request if the resources are already set to the right value?

1. Fluentd pods are getting deleted and recreated by daemonset-controller when the scaler is enabled (as was also seen in https://github.com/kubernetes/kubernetes/issues/60500#issuecomment-373001797). Why this is happening? One thing to note here - all those delete calls are preceded by PUT pod-status calls from respective kubelets (but maybe that’s expected).

cc @kubernetes/sig-instrumentation-bugs @crassirostris @liggitt /priority critical-urgent /assign @x13n