argo-rollouts: Operation cannot be fulfilled on rollouts.argoproj.io: the object has been modified

Summary

When creating a new rollout we sometimes see it get stuck and never progress. Logs show it failing to update due to a resource conflict. It retries but always get the same conflict error, even though it re-queues after each failure.

Related: #1775

My guess is that the revision update at: https://github.com/argoproj/argo-rollouts/blob/acf805423c48f0e037e6e5686e6d164465f33c51/rollout/sync.go#L92-L94 succeeds, and then causes the status update at: https://github.com/argoproj/argo-rollouts/blob/acf805423c48f0e037e6e5686e6d164465f33c51/rollout/sync.go#L104-L108 to get the conflict. Both log the same error so it’s hard to tell for sure.

I’m happy to work on a PR for this but I’m new to Argo and the code. Normally I’d return after the revision update and do the status update on the next reconcile but it looks like the revision is incremented each time syncReplicaSetRevision() runs? Alternatively, after the revision update, re-fetch the ReplicaSet to make sure we have the latest version for the status update.

Any thoughts or guidance appreciated!

Diagnostics

What version of Argo Rollouts are you running? v1.0.2

time="2022-03-09T17:57:31Z" level=info msg="Defaulting .spec.replica to 1" generation=1 namespace=testns resourceVersion=4389783140 rollout=app
time="2022-03-09T17:57:31Z" level=info msg="Started syncing rollout" generation=1 namespace=testns resourceVersion=4389783140 rollout=app
time="2022-03-09T17:57:35Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389783922 rollout=app
time="2022-03-09T17:57:35Z" level=info msg="Updating replica set 'app-746777884f' revision from 0 to 1" namespace=testns rollout=app
time="2022-03-09T17:57:35Z" level=info msg="Created ReplicaSet app-746777884f" namespace=testns rollout=app
time="2022-03-09T17:57:39Z" level=info msg="Created ReplicaSet app-746777884f (revision 1) with size 1" event_reason=NewReplicaSetCreated namespace=testns rollout=app
time="2022-03-09T17:57:39Z" level=info msg="Rollout updated to revision 1" event_reason=RolloutUpdated namespace=testns rollout=app
time="2022-03-09T17:57:44Z" level=info msg="rollout syncHandler queue retries: 7 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:57:44Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:57:44Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389783922 rollout=app time_ms=8391.492779
time="2022-03-09T17:57:44Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389783922 rollout=app
time="2022-03-09T17:57:44Z" level=info msg="persisted to informer" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:57:44Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:57:48Z" level=info msg="rollout syncHandler queue retries: 8 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:57:48Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:57:48Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4191.9283700000005
time="2022-03-09T17:57:48Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:57:48Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:57:48Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:57:52Z" level=info msg="rollout syncHandler queue retries: 9 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:57:52Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:57:52Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4200.384081
time="2022-03-09T17:57:52Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:57:52Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:57:52Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:57:57Z" level=info msg="rollout syncHandler queue retries: 10 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:57:57Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:57:57Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4599.406142999999
time="2022-03-09T17:57:57Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:57:57Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:57:57Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:01Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:01Z" level=info msg="rollout syncHandler queue retries: 11 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:58:01Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:01Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4598.759919
time="2022-03-09T17:58:01Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:01Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:05Z" level=info msg="rollout syncHandler queue retries: 13 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:58:05Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:05Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4205.47663
time="2022-03-09T17:58:05Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:05Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:05Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:10Z" level=info msg="rollout syncHandler queue retries: 14 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:58:10Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:10Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4194.796269
time="2022-03-09T17:58:10Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:10Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:10Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:14Z" level=info msg="rollout syncHandler queue retries: 15 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:58:14Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:14Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4200.356353
time="2022-03-09T17:58:14Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:14Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:18Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:22Z" level=info msg="rollout syncHandler queue retries: 16 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:58:22Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:22Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4408.043619
time="2022-03-09T17:58:22Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:22Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:32Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:37Z" level=info msg="rollout syncHandler queue retries: 17 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:58:37Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:37Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4398.639260000001
time="2022-03-09T17:58:37Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:37Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:47Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:51Z" level=info msg="rollout syncHandler queue retries: 18 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:58:51Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:58:51Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4200.2296719999995
time="2022-03-09T17:58:51Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:58:51Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:01Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:05Z" level=info msg="rollout syncHandler queue retries: 19 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:59:05Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:05Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4196.487878
time="2022-03-09T17:59:05Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:05Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:15Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:20Z" level=info msg="rollout syncHandler queue retries: 20 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:59:20Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:20Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4600.653727
time="2022-03-09T17:59:20Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:20Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:30Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:34Z" level=info msg="rollout syncHandler queue retries: 21 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:59:34Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:34Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4199.269765999999
time="2022-03-09T17:59:34Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:34Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:44Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:48Z" level=info msg="rollout syncHandler queue retries: 22 : key \"testns/app\"" namespace=testns rollout=app
time="2022-03-09T17:59:48Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:48Z" level=info msg="Reconciliation completed" generation=2 namespace=testns resourceVersion=4389784599 rollout=app time_ms=4198.823166
time="2022-03-09T17:59:48Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" generation=2 namespace=testns resourceVersion=4389784599 rollout=app
time="2022-03-09T17:59:48Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"app\": the object has been modified; please apply your changes to the latest version and try again" namespace=testns rollout=app
time="2022-03-09T17:59:58Z" level=info msg="Started syncing rollout" generation=2 namespace=testns resourceVersion=4389784599 rollout=app

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.