argo-rollouts: Rollout stuck issue
Checklist:
- I’ve included steps to reproduce the bug.
- I’ve included the version of argo rollouts.
Describe the bug
We seem to have faced the issue described in:
https://github.com/argoproj/argo-rollouts/pull/3272 https://github.com/argoproj/argo-rollouts/pull/3257 https://github.com/argoproj/argo-rollouts/issues/3256
To mitigate the issue we’ve deployed master branch build from revision:
* 180b4bff (HEAD -> master, origin/master, origin/HEAD) chore(deps): bump github.com/evanphx/json-patch/v5 from 5.7.0 to 5.8.0 (#3309)
However it does not solve the issue.
To Reproduce
Expected behavior
Screenshots
Version
* 180b4bff (HEAD -> master, origin/master, origin/HEAD) chore(deps): bump github.com/evanphx/json-patch/v5 from 5.7.0 to 5.8.0 (#3309)
Logs
rollout controller:
# Paste the logs from the rollout controller
time="2024-01-18T13:05:34Z" level=error msg="roCtx.reconcile err error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal
time="2024-01-18T13:05:34Z" level=info msg="Reconciliation completed" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal time_ms=38.83042
time="2024-01-18T13:05:34Z" level=error msg="rollout syncHandler error: error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again" namespace=app-namespace rollout=application-internal
time="2024-01-18T13:05:34Z" level=info msg="rollout syncHandler queue retries: 258 : key \"app-namespace/application-internal\"" namespace=app-namespace rollout=application-internal
time="2024-01-18T13:05:34Z" level=error msg="error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"application-internal-6fc65dcfc6\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"
time="2024-01-18T13:05:44Z" level=info msg="Started syncing rollout" generation=757 namespace=app-namespace resourceVersion=4332321110 rollout=application-internal
argo rollouts:
$ kubectl argo rollouts get rollout application-internal -n app-namespace
Name: application-internal
Namespace: app-namespace
Status: ◌ Progressing
Message: more replicas need to be updated
Strategy: Canary
Step: 0/3
SetWeight: 10
ActualWeight: 0
Images: eu.gcr.io/.../application-internal:6939469
eu.gcr.io/.../application-internal:afeee0f (stable)
Replicas:
Desired: 4
Current: 4
Updated: 0
Ready: 4
Available: 4
NAME KIND STATUS AGE INFO
⟳ application-internal Rollout ◌ Progressing 134d
├──# revision:717
│ └──⧉ application-internal-546856bb5c ReplicaSet • ScaledDown 52m canary
├──# revision:716
│ └──⧉ application-internal-7fc96c849c ReplicaSet • ScaledDown 122m
├──# revision:715
│ ├──⧉ application-internal-575b7d5fc6 ReplicaSet ✔ Healthy 161m
│ │ └──□ application-internal-575b7d5fc6-fxbzf Pod ✔ Running 161m ready:2/2
│ └──α application-internal-575b7d5fc6-715-2 AnalysisRun ✔ Successful 148m ✔ 1
├──# revision:714
│ ├──⧉ application-internal-6fc65dcfc6 ReplicaSet ✔ Healthy 177m stable
│ │ ├──□ application-internal-6fc65dcfc6-fxgnb Pod ✔ Running 177m ready:2/2
│ │ └──□ application-internal-6fc65dcfc6-ll8xg Pod ✔ Running 164m ready:2/2
│ └──α application-internal-6fc65dcfc6-714-2 AnalysisRun ✔ Successful 165m ✔ 1
└──# revision:713
└──⧉ application-internal-5f8bb4b544 ReplicaSet • ScaledDown 179m
rs:
$ kubectl -n app-namespace get rs | grep application-internal
application-internal-546856bb5c 0 0 0 52m # rollout.argoproj.io/revision: "717"
application-internal-7fc96c849c 0 0 0 122m # rollout.argoproj.io/revision: "716"
application-internal-575b7d5fc6 1 1 1 161m # rollout.argoproj.io/revision: "715"
application-internal-6fc65dcfc6 2 2 2 177m # rollout.argoproj.io/revision: "714"
application-internal-5f8bb4b544 0 0 0 3h # rollout.argoproj.io/revision: "713"
happy to help with debugging, deploying a custom version, etc.
Workarond
We’ve restarted deploy/argo-rollouts.
After controller restart:
$ kubectl argo rollouts get rollout application-internal -n app-namespace
Name: application-internal
Namespace: app-namespace
Status: ◌ Progressing
Message: more replicas need to be updated
Strategy: Canary
Step: 0/3
SetWeight: 10
ActualWeight: 0
Images: eu.gcr.io/.../application-internal:1ba0f90 (canary)
eu.gcr.io/.../application-internal:afeee0f (stable)
Replicas:
Desired: 4
Current: 4
Updated: 0
Ready: 4
Available: 4
NAME KIND STATUS AGE INFO
⟳ application-internal Rollout ◌ Progressing 134d
├──# revision:717
│ └──⧉ application-internal-546856bb5c ReplicaSet ◌ Progressing 54m canary
│ └──□ application-internal-546856bb5c-99xp7 Pod ◌ Init:0/2 0s ready:0/2
├──# revision:716
│ └──⧉ application-internal-7fc96c849c ReplicaSet • ScaledDown 123m
├──# revision:715
│ ├──⧉ application-internal-575b7d5fc6 ReplicaSet • ScaledDown 163m
│ │ └──□ application-internal-575b7d5fc6-fxbzf Pod ◌ Terminating 163m ready:2/2
│ └──α application-internal-575b7d5fc6-715-2 AnalysisRun ✔ Successful 149m ✔ 1
├──# revision:714
│ ├──⧉ application-internal-6fc65dcfc6 ReplicaSet ✔ Healthy 178m stable
│ │ ├──□ application-internal-6fc65dcfc6-fxgnb Pod ✔ Running 178m ready:2/2
│ │ ├──□ application-internal-6fc65dcfc6-ll8xg Pod ✔ Running 166m ready:2/2
│ │ ├──□ application-internal-6fc65dcfc6-dp8m8 Pod ✔ Running 31s ready:2/2
│ │ └──□ application-internal-6fc65dcfc6-jhcgr Pod ✔ Running 31s ready:2/2
│ └──α application-internal-6fc65dcfc6-714-2 AnalysisRun ✔ Successful 166m ✔ 1
└──# revision:713
└──⧉ application-internal-5f8bb4b544 ReplicaSet • ScaledDown 3h1m
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 97
- Comments: 39 (7 by maintainers)
Yup I have also found this there a few places where we clobber the in memory state of the replicaset. I do suspect this is somewhat the root cause. The one thing that bothers me though is why the retry reconcile does not correct it and instead it gets stuck. I think the issues has been there a while but it seems to have gotten worse with recent versions. This is going to become one of my top priorities to figure out.
We have possibly found the root cause of these issues would love feed back on v1.8.0-rc2+ we are running the rc2 right now in production and have been for a bit now things are looking good but we also did not hit the issue super frequently.
This issue included information on a lot of different bugs. The Intuit team encountered one bug today, so I’ll share the details of that particular problem:
We could build a mechanism into Argo Rollouts to re-list ReplicaSets occasionally and clear stale data. That would be a significant refactor.
Update: we’ve found that making an insignificant edit on old ReplicaSets instead of deleting them is a safer and equally effective way to clear the issue. I just added annotation
test: test.@zachaller Thank you so much for working on this! I tested with both
v1.8.0-rc2andv1.8.2and unfortunately the issue persists in both cases.I updated the controller to v1.6.5 and this issue still occurs. https://github.com/argoproj/argo-rollouts/issues/3080#issuecomment-1835809731
For now, we run a job to restart the controller every hour, but we still get
ProgressDeadlineExceeded: ReplicaSet ... has timed out progressingerrors.V1.6.6 was released it fixes one reproducible version of a stuck rollout people might want to give it a try.
I’ve seen the issue with 1.6.4 where the newest replicaset have a relatively small number of current and desired replicas (but more than minimum from KEDA HPA), while an older replicaset was getting desired replicas to the max, but was stuck at 0 replicas. It actually used to work for some time after doing a rollout with scaling as expected, but something broke later causing the behavior above. Doing a new rollout mitigated the issue, but it feels like playing a lottery.
It seems like one of two things:
Maybe the warning about “the object has been modified; please apply your changes to the latest version and try again” is just a symptom of propagating desired replicas to the older replica set.
I’m facing with similar issue where my Rollouts with HPA stack on Progressing status permanently in v1.6.2. (This occurs randomly, and I’ve not tested v1.6.5. yet) In my case, we have some strange Rollouts status as following:
Logs
Rollouts is fully promoted and new version released, but stacks on Progressing status because of inconsistent replicaset.count. Intuitively, I suspect HPA should relevent with this issue. Unlike #3256, the pod template has not been changed.
Additional info
We’re seeing this issue in
v1.8.1+1ad2c6a@jccastillocano conflicts are generally not an issue and a normal k8s pattern, the conflict here is on the rollout resource. The patch in the issue is just on conflicts related to replicasets. This is to insure that we can always update the scale of the replicaset and avoids conflict loops. We have not seen any loops on conflicts on Rollout resources.
+1 I am trying a cronJob to restart it daily to see if it helps to reduce the frequency of the issue
Yes, sure. We’d be happy to deploy a custom version to gather more information about this issue.