argo-rollouts: Using canary w/ dynamic stable scale causes failed requests at end of rollout
Describe the bug
Hello,
I recently enabled Istio-based traffic routing and the dynamicStableScale feature on my application. The feature works as described except it is causing a brief period of failed requests at the end of the rollout.
My spec is roughly as follows:
spec:
strategy:
canary:
trafficRouting: { istio: { ... } }
# Scale down stable replica set as the traffic weight increases to canary
dynamicStableScale: true
# Run background analysis throughout rollout
analysis: ...
steps:
# Create 1 canary replica for testing purposes
- setCanaryScale: { replicas: 1 }
# Run in-line analysis against canary replica
- analysis: ...
# Returns to the default behavior of matching the canary traffic weight
- setCanaryScale: { matchTrafficWeight: true }
# Scale up the canary
- setWeight: 25
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 5m }
- setWeight: 75
- pause: { duration: 5m }
- setWeight: 100
# Pause post-deployment to allow background analysis to evaluate 100% weight
- pause: { duration: 5m }
What I’m repeatedly observing is that as soon as the canary weight is increased to 100%, the last stable replica will be scaled down and clients to the service will encounter “no healthy upstream” Istio errors for approximately 30 seconds. I believe this occurs at a rate proportional to the most recent weight increase, or about 25% in the above example. I suspect the 30 seconds may be partly due to Istio propagation delay but propagation is usually within a few seconds.
Is this intended behavior? Is there anything I can configure to allow my stable endpoint sufficient time to drain before fully scaling it down?
More specifically, I want the behavior of dynamicStableScale (so that I am not running 2x my baseline number of replicas), but I want to ensure that the stable replica set is left with at least 1 healthy/ready replica for a draining period after its traffic weight is set to 0%.
Configuration options I’ve looked at but don’t seem to quite fit:
minReadySeconds(defaults to 0, so this may help, but what if canary reaches full replica size in a previous step due to rounding?)maxUnavailable(defaults to 1, so does not appear to be used after going to 100%)scaleDownDelaySeconds(default is 30s, but errors appear immediately after going to 100%)minPodsPerReplicaSet(defaults to 1, so does not appear to be used after going to 100%)
Version
1.7.0
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 15 (5 by maintainers)
Same problem here.
We use Istio with DestinationRule subsets, we don’t use
dynamicStableScaleand we don’t set pause after 100% as well.As soon as the traffic shifts to 100% some clients receive
no healthy upstream.I also saw it on follow up deployments as well.
– Kind regards, Wietse
On Wed, 17 Jul 2024 at 19:58 Bennett Lynch @.***> wrote:
Might possibly be fixed by https://github.com/argoproj/argo-rollouts/pull/3602 as well.
After more testing, this issue only seems to occur when both:
dynamicStableScaleis enabled