argo-rollouts: Using canary w/ dynamic stable scale causes failed requests at end of rollout

Describe the bug

Hello,

I recently enabled Istio-based traffic routing and the dynamicStableScale feature on my application. The feature works as described except it is causing a brief period of failed requests at the end of the rollout.

My spec is roughly as follows:

spec:
  strategy:
    canary:
      trafficRouting: { istio: { ... } }
      # Scale down stable replica set as the traffic weight increases to canary
      dynamicStableScale: true
      # Run background analysis throughout rollout
      analysis: ...
      steps:
        # Create 1 canary replica for testing purposes
        - setCanaryScale: { replicas: 1 }
        # Run in-line analysis against canary replica
        - analysis: ...
        # Returns to the default behavior of matching the canary traffic weight
        - setCanaryScale: { matchTrafficWeight: true }
        # Scale up the canary
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 75
        - pause: { duration: 5m }
        - setWeight: 100
        # Pause post-deployment to allow background analysis to evaluate 100% weight
        - pause: { duration: 5m }

What I’m repeatedly observing is that as soon as the canary weight is increased to 100%, the last stable replica will be scaled down and clients to the service will encounter “no healthy upstream” Istio errors for approximately 30 seconds. I believe this occurs at a rate proportional to the most recent weight increase, or about 25% in the above example. I suspect the 30 seconds may be partly due to Istio propagation delay but propagation is usually within a few seconds.

Is this intended behavior? Is there anything I can configure to allow my stable endpoint sufficient time to drain before fully scaling it down?

More specifically, I want the behavior of dynamicStableScale (so that I am not running 2x my baseline number of replicas), but I want to ensure that the stable replica set is left with at least 1 healthy/ready replica for a draining period after its traffic weight is set to 0%.

Configuration options I’ve looked at but don’t seem to quite fit:

  1. minReadySeconds (defaults to 0, so this may help, but what if canary reaches full replica size in a previous step due to rounding?)
  2. maxUnavailable (defaults to 1, so does not appear to be used after going to 100%)
  3. scaleDownDelaySeconds (default is 30s, but errors appear immediately after going to 100%)
  4. minPodsPerReplicaSet (defaults to 1, so does not appear to be used after going to 100%)

Version

1.7.0


Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 5
  • Comments: 15 (5 by maintainers)

Most upvoted comments

Same problem here.

We use Istio with DestinationRule subsets, we don’t use dynamicStableScale and we don’t set pause after 100% as well.

As soon as the traffic shifts to 100% some clients receive no healthy upstream.

I also saw it on follow up deployments as well.

– Kind regards, Wietse

On Wed, 17 Jul 2024 at 19:58 Bennett Lynch @.***> wrote:

Thanks @wmuizelaar https://github.com/wmuizelaar. Is that only for “initial deployments” as the title suggests? I’m encountering this issue on repeated deployments with Istio.

— Reply to this email directly, view it on GitHub https://github.com/argoproj/argo-rollouts/issues/3681#issuecomment-2233898364, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6ZXVAGGOMFU6BTZUGMSZ3ZM2WFLAVCNFSM6AAAAABJ64PHFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTHA4TQMZWGQ . You are receiving this because you were mentioned.Message ID: @.***>

After more testing, this issue only seems to occur when both:

  1. dynamicStableScale is enabled
  2. there is a pause step after setting the canary weight to 100%