argo-rollouts: Argo rollouts will scale down stable when canary is missing pods

Summary

Argo scaled down the stable RS even while the new replica set is not fully available. In this case, it scaled from 110 pods down to 0 but didn’t update the routing until the new RS was 100% ready.
Argo shouldn’t scale down the stable RS until it switches the to use the new canary as stable.
Argo shouldn’t scale down the stable RS until the canary RS is ready.
Diagnostics

What version of Argo Rollouts are you running?
1.2.0
time="2022-05-24T07:57:11Z" level=info msg="Previous weights: &TrafficWeights{Canary:WeightDestination{Weight:100,ServiceName:api-v2-canary,PodTemplateHash:5bb4f8b6b6,},Stable:WeightDestination{Weight:0,ServiceName:api-v2-stable,PodTemplateHash:779d9dfb64,},Additional:[]WeightDestination{},Verified:nil,}" namespace=cs-team rollout=api-v2
--
  |   | time="2022-05-24T07:57:11Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:11Z" level=info msg="Enqueueing parent of cs-team/api-v2-779d9dfb64: Rollout cs-team/api-v2"
  |   | time="2022-05-24T07:57:11Z" level=info msg="Enqueueing parent of cs-team/api-v2-779d9dfb64: Rollout cs-team/api-v2"
  |   | time="2022-05-24T07:57:11Z" level=info msg="Enqueueing parent of cs-team/api-v2-779d9dfb64: Rollout cs-team/api-v2"
  |   | time="2022-05-24T07:57:11Z" level=info msg="Set 'scale-down-deadline' annotation on 'api-v2-779d9dfb64' to 2022-05-24T07:57:41Z (30s)" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:12Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:12Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:12Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:12Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:12Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:14Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:14Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:14Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:33Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:33Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:33Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:34Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:34Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:34Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:34Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:34Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:35Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:36Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:37Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:38Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:39Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:39Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:40Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:40Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:40Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:40Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:40Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:40Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:40Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:40Z" level=info msg="RS 'api-v2-779d9dfb64' has not reached the scaleDownTime" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:41Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:41Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:41Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:41Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:57:41Z" level=info msg="Found 110 available pods in old RS cs-team/api-v2-779d9dfb64" namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:41Z" level=info msg="Enqueueing parent of cs-team/api-v2-779d9dfb64: Rollout cs-team/api-v2"
  |   | time="2022-05-24T07:57:41Z" level=info msg="Enqueueing parent of cs-team/api-v2-779d9dfb64: Rollout cs-team/api-v2"
  |   | time="2022-05-24T07:57:41Z" level=info msg="Enqueueing parent of cs-team/api-v2-779d9dfb64: Rollout cs-team/api-v2"
  |   | time="2022-05-24T07:57:41Z" level=info msg="Scaled down ReplicaSet api-v2-779d9dfb64 (revision 3292) from 110 to 0" event_reason=ScalingReplicaSet namespace=cs-team rollout=api-v2
  |   | time="2022-05-24T07:57:41Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"cs-team\", Name:\"api-v2\", UID:\"2a76d2a7-e138-4113-b3aa-eb1ec96011cd\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"1213092265\", FieldPath:\"\"}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down ReplicaSet api-v2-779d9dfb64 (revision 3292) from 110 to 0"

....



time="2022-05-24T07:59:48Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable Show context
--
  |   | time="2022-05-24T07:59:59Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:59:59Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:59:59Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:59:59Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:59:59Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T07:59:59Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T08:00:08Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T08:00:08Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T08:00:08Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T08:00:08Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T08:00:08Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T08:00:08Z" level=info msg="delaying service switch from 779d9dfb64 to 5bb4f8b6b6: ReplicaSet not fully available" namespace=cs-team rollout=api-v2 service=api-v2-stable
  |   | time="2022-05-24T08:00:25Z" level=info msg="Event(v1.ObjectReference{Kind:\"Rollout\", Namespace:\"cs-team\", Name:\"api-v2\", UID:\"2a76d2a7-e138-4113-b3aa-eb1ec96011cd\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"1213103929\", FieldPath:\"\"}): type: 'Normal' reason: 'SwitchService' Switched selector for service 'api-v2-stable' from '779d9dfb64' to '5bb4f8b6b6'"
  |   | time="2022-05-24T08:00:25Z" level=info msg="Switched selector for service 'api-v2-stable' from '779d9dfb64' to '5bb4f8b6b6'" event_reason=SwitchService namespace=cs-team rollout=api-v2
I’ve tried to brief on these logs, but I can get the full logs if you would like.
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 19
Comments: 20 (10 by maintainers)
Commits related to this issue

fix: return an error when we cannot swap the replicaset hashes fixes #2050 — committed to jandersen-plaid/argo-rollouts by jandersen-plaid 2 years ago
fix: return an error when we cannot swap the replicaset hashes fixes #2050 Signed-off-by: Jack Andersen <jandersen@plaid.com> — committed to jandersen-plaid/argo-rollouts by jandersen-plaid 2 years ago
fix: return an error when we cannot swap the replicaset hashes fixes #2050 Signed-off-by: Jack Andersen <jandersen@plaid.com> — committed to jandersen-plaid/argo-rollouts by jandersen-plaid 2 years ago
fix: return an error when we cannot swap the replicaset hashes fixes #2050 Signed-off-by: Jack Andersen <jandersen@plaid.com> — committed to jandersen-plaid/argo-rollouts by jandersen-plaid 2 years ago
fix: return an error when we cannot swap the replicaset hashes fixes #2050 Signed-off-by: Jack Andersen <jandersen@plaid.com> — committed to jandersen-plaid/argo-rollouts by jandersen-plaid 2 years ago
fix: return an error when we cannot swap the replicaset hashes fixes #2050 Signed-off-by: Jack Andersen <jandersen@plaid.com> — committed to jandersen-plaid/argo-rollouts by jandersen-plaid 2 years ago
Most upvoted comments

@raxod502-plaid @jandersen-plaid @MarkSRobinson apologies: was doing a lot of avatar clicking to see who was officially on the project and mistook Mark for one of them.
jstewart612 on Dec 8, 2022
@zachaller I built it and pushed it to our internal repo. We’re testing it out on the testing cluster right now.
MarkSRobinson on Dec 6, 2022