argo-rollouts: Rollout stuck issue

Checklist:

  • I’ve included steps to reproduce the bug.
  • I’ve included the version of argo rollouts.

Describe the bug

We seem to have faced the issue described in:

https://github.com/argoproj/argo-rollouts/pull/3272 https://github.com/argoproj/argo-rollouts/pull/3257 https://github.com/argoproj/argo-rollouts/issues/3256

To mitigate the issue we’ve deployed master branch build from revision:

* 180b4bff (HEAD -> master, origin/master, origin/HEAD) chore(deps): bump github.com/evanphx/json-patch/v5 from 5.7.0 to 5.8.0 (#3309)

However it does not solve the issue.

To Reproduce

Expected behavior

Screenshots

Version

* 180b4bff (HEAD -> master, origin/master, origin/HEAD) chore(deps): bump github.com/evanphx/json-patch/v5 from 5.7.0 to 5.8.0 (#3309)

Logs

rollout controller:

# Paste the logs from the rollout controller

time="2024-01-18T13:05:34Z" level=error msg="roCtx.reconcile err error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal

time="2024-01-18T13:05:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal time_ms=38.83042

time="2024-01-18T13:05:34Z" level=error msg="rollout syncHandler error: error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again" namespace=app-namespace rollout=application-internal

time="2024-01-18T13:05:34Z" level=info msg="rollout syncHandler queue retries: 258 : key \"app-namespace/application-internal\"" namespace=app-namespace rollout=application-internal

time="2024-01-18T13:05:34Z" level=error msg="error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"

time="2024-01-18T13:05:44Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal

argo rollouts:

$ kubectl  argo rollouts get rollout application-internal  -n app-namespace
Name:            application-internal
Namespace:       app-namespace
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/3
  SetWeight:     10
  ActualWeight:  0
Images:          eu.gcr.io/.../application-internal:6939469
                 eu.gcr.io/.../application-internal:afeee0f (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       0
  Ready:         4
  Available:     4

NAME                                         KIND         STATUS         AGE   INFO
⟳ application-internal                            Rollout      ◌ Progressing  134d
├──# revision:717
│  └──⧉ application-internal-546856bb5c           ReplicaSet   • ScaledDown   52m   canary
├──# revision:716
│  └──⧉ application-internal-7fc96c849c           ReplicaSet   • ScaledDown   122m
├──# revision:715
│  ├──⧉ application-internal-575b7d5fc6           ReplicaSet   ✔ Healthy      161m
│  │  └──□ application-internal-575b7d5fc6-fxbzf  Pod          ✔ Running      161m  ready:2/2
│  └──α application-internal-575b7d5fc6-715-2     AnalysisRun  ✔ Successful   148m  ✔ 1
├──# revision:714
│  ├──⧉ application-internal-6fc65dcfc6           ReplicaSet   ✔ Healthy      177m  stable
│  │  ├──□ application-internal-6fc65dcfc6-fxgnb  Pod          ✔ Running      177m  ready:2/2
│  │  └──□ application-internal-6fc65dcfc6-ll8xg  Pod          ✔ Running      164m  ready:2/2
│  └──α application-internal-6fc65dcfc6-714-2     AnalysisRun  ✔ Successful   165m  ✔ 1
└──# revision:713
   └──⧉ application-internal-5f8bb4b544           ReplicaSet   • ScaledDown   179m

rs:

$ kubectl -n app-namespace  get rs | grep application-internal
application-internal-546856bb5c   0         0         0       52m  # rollout.argoproj.io/revision: "717"
application-internal-7fc96c849c   0         0         0       122m # rollout.argoproj.io/revision: "716"
application-internal-575b7d5fc6   1         1         1       161m # rollout.argoproj.io/revision: "715"
application-internal-6fc65dcfc6   2         2         2       177m # rollout.argoproj.io/revision: "714"
application-internal-5f8bb4b544   0         0         0       3h # rollout.argoproj.io/revision: "713"

happy to help with debugging, deploying a custom version, etc.

Workarond

We’ve restarted deploy/argo-rollouts. After controller restart:

$ kubectl  argo rollouts get rollout application-internal  -n app-namespace
Name:            application-internal
Namespace:       app-namespace
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/3
  SetWeight:     10
  ActualWeight:  0
Images:          eu.gcr.io/.../application-internal:1ba0f90 (canary)
                 eu.gcr.io/.../application-internal:afeee0f (stable)
Replicas:
  Desired:       4
  Current:       4
  Updated:       0
  Ready:         4
  Available:     4

NAME                                         KIND         STATUS         AGE   INFO
⟳ application-internal                            Rollout      ◌ Progressing  134d
├──# revision:717
│  └──⧉ application-internal-546856bb5c           ReplicaSet   ◌ Progressing  54m   canary
│     └──□ application-internal-546856bb5c-99xp7  Pod          ◌ Init:0/2     0s    ready:0/2
├──# revision:716
│  └──⧉ application-internal-7fc96c849c           ReplicaSet   • ScaledDown   123m
├──# revision:715
│  ├──⧉ application-internal-575b7d5fc6           ReplicaSet   • ScaledDown   163m
│  │  └──□ application-internal-575b7d5fc6-fxbzf  Pod          ◌ Terminating  163m  ready:2/2
│  └──α application-internal-575b7d5fc6-715-2     AnalysisRun  ✔ Successful   149m  ✔ 1
├──# revision:714
│  ├──⧉ application-internal-6fc65dcfc6           ReplicaSet   ✔ Healthy      178m  stable
│  │  ├──□ application-internal-6fc65dcfc6-fxgnb  Pod          ✔ Running      178m  ready:2/2
│  │  ├──□ application-internal-6fc65dcfc6-ll8xg  Pod          ✔ Running      166m  ready:2/2
│  │  ├──□ application-internal-6fc65dcfc6-dp8m8  Pod          ✔ Running      31s   ready:2/2
│  │  └──□ application-internal-6fc65dcfc6-jhcgr  Pod          ✔ Running      31s   ready:2/2
│  └──α application-internal-6fc65dcfc6-714-2     AnalysisRun  ✔ Successful   166m  ✔ 1
└──# revision:713
   └──⧉ application-internal-5f8bb4b544           ReplicaSet   • ScaledDown   3h1m

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 97
  • Comments: 39 (7 by maintainers)

Most upvoted comments

hi guys we also experienced some issues that canary rollout stucked, we couldn’t reproduce steadily but what I found when I added a couple more logs in argo rollouts, it seems like there is race condition betweem the syncReplicasOnly and rolloutCanary that they are updating the c.newRS at the same time (and potentially together with reconcileEphemeralMetadata which is syncing c.newRS to kubernetes at the same time), so the value in c.newRS could be different than the actual replicaSet in the kubernetes.

Yup I have also found this there a few places where we clobber the in memory state of the replicaset. I do suspect this is somewhat the root cause. The one thing that bothers me though is why the retry reconcile does not correct it and instead it gets stuck. I think the issues has been there a while but it seems to have gotten worse with recent versions. This is going to become one of my top priorities to figure out.

We have possibly found the root cause of these issues would love feed back on v1.8.0-rc2+ we are running the rc2 right now in production and have been for a bit now things are looking good but we also did not hit the issue super frequently.

This issue included information on a lot of different bugs. The Intuit team encountered one bug today, so I’ll share the details of that particular problem:

When a Rollout starts, Argo kicks off a timeout counter. We require that, before that timer expires, the number replicas running in the new ReplicaSet is equal to the sum of all replicas running in all ReplicaSets (i.e. everything is scaled to where it should be scaled).

Argo relies on a cache of all ReplicaSet objects to calculate how many replicas each is running. That cache is maintained by handling an event stream from the Kubernetes API.

If an event fails to be delivered, Argo Rollouts can retain a stale replica count for an old ReplicaSet. In this case, it retained a replica count of 1 for the old ReplicaSet. So Argo Rollouts believed 4 Pods were running instead of the desired 3. There is no internal mechanism to clear this stale data.

Once the progressing deadline is exceeded due to the stale data, we mark the Rollout as Degraded. It will stay that way until the stale data is cleared.

Workarounds include:

  • Restart the Rollouts controller. This will re-list all ReplicaSets and get up-to-date data.
  • Delete old ReplicaSets (starting with the oldest and moving to the newest non-active ReplicaSet) until the Rollout becomes healthy. By deleting the stale ReplicaSet, you force a re-calculation of running replicas excluding the stale ReplicaSet. This strategy can be problematic. If you delete the second most-recent ReplicaSet, you complicate the rollback process if an issue is encountered with the newest revision.

We could build a mechanism into Argo Rollouts to re-list ReplicaSets occasionally and clear stale data. That would be a significant refactor.

Update: we’ve found that making an insignificant edit on old ReplicaSets instead of deleting them is a safer and equally effective way to clear the issue. I just added annotation test: test.

@zachaller Thank you so much for working on this! I tested with both v1.8.0-rc2 and v1.8.2 and unfortunately the issue persists in both cases.

I updated the controller to v1.6.5 and this issue still occurs. https://github.com/argoproj/argo-rollouts/issues/3080#issuecomment-1835809731

For now, we run a job to restart the controller every hour, but we still get ProgressDeadlineExceeded: ReplicaSet ... has timed out progressing errors.

V1.6.6 was released it fixes one reproducible version of a stuck rollout people might want to give it a try.

I’ve seen the issue with 1.6.4 where the newest replicaset have a relatively small number of current and desired replicas (but more than minimum from KEDA HPA), while an older replicaset was getting desired replicas to the max, but was stuck at 0 replicas. It actually used to work for some time after doing a rollout with scaling as expected, but something broke later causing the behavior above. Doing a new rollout mitigated the issue, but it feels like playing a lottery.

It seems like one of two things:

  • Rollout started propagating desired replicas to a wrong, older replica set. But since it was completed, there’s been some check preventing old replica set to scale.
  • Rollout propagated desired replicas to an older replica set to begin with, but it used to work until rollout wasn’t somehow marked as completed.

Maybe the warning about “the object has been modified; please apply your changes to the latest version and try again” is just a symptom of propagating desired replicas to the older replica set.

I’m facing with similar issue where my Rollouts with HPA stack on Progressing status permanently in v1.6.2. (This occurs randomly, and I’ve not tested v1.6.5. yet) In my case, we have some strange Rollouts status as following:

status:
  HPAReplicas: 13
  availableReplicas: 13
  blueGreen: {}
  canary: {} 
  currentPodHash: 5f9cf5d68b
  currentStepHash: d68d4f7d8
  currentStepIndex: 2
  message: old replicas are pending termination
  observedGeneration: '8654'
  phase: Progressing
  readyReplicas: 11
  replicas: 11
  stableRS: 5f9cf5d68b
  updatedReplicas: 10

Logs

time="2024-01-30T17:16:05Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on replicasets.apps \"app-6b76b6dd98\": the object has been modified; please apply your changes to the latest version and try again" namespace=apps rollout=app
time="2024-01-30T17:16:05Z" level=info msg="rollout syncHandler queue retries: 1865 : key \"apps/app\"" namespace=apps rollout=app

Rollouts is fully promoted and new version released, but stacks on Progressing status because of inconsistent replicaset.count. Intuitively, I suspect HPA should relevent with this issue. Unlike #3256, the pod template has not been changed.

Additional info

  • After restart rollout controller, Progressing status changes to Sucessful, so everything works fine!

We’re seeing this issue in v1.8.1+1ad2c6a

@jccastillocano conflicts are generally not an issue and a normal k8s pattern, the conflict here is on the rollout resource. The patch in the issue is just on conflicts related to replicasets. This is to insure that we can always update the scale of the replicaset and avoids conflict loops. We have not seen any loops on conflicts on Rollout resources.

+1 I am trying a cronJob to restart it daily to see if it helps to reduce the frequency of the issue

Yes, sure. We’d be happy to deploy a custom version to gather more information about this issue.