argo-rollouts: Rollout got stuck

Checklist:

  • I’ve included steps to reproduce the bug.
  • I’ve included the version of argo rollouts.

Describe the bug From time to time we’re facing issue that when we’re deploying new version and argo rollout need to rollout new pods, it just creating the new RS and then got stuck, when promoting manually the rollouts or restarting argo-rollout controller it fix the issue.

To Reproduce N/A

Expected behavior Rollout will proceed with deployment of changes even if something else edited the rollout object

Version 1.3.2

Logs Attached both full log of controller and for the rollouts called my-first-app: full-controller-logs.csv my-first-app-logs.csv


Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 18
  • Comments: 40 (1 by maintainers)

Most upvoted comments

Hey, coming here to raise that issue and try to open the discussion again. We face the exact same issue with HPA.

@triplewy This seems to happen with us as well with the HPA.

@zachaller So if I understand correctly, if we have an HPA that scales a rollout, this will come into conflict with the argo-rollouts controller? Because our setup is simple where we have a rollout that can be scaled by an HPA and we occassionaly get the error: rollout syncHandler error: error updating replicaset : Operation cannot be fulfilled on replicasets.apps \"xxx": the object has been modified; please apply your changes to the latest version and try again" namespace=xxx rollout=xxx

also one other good bit of info is this log line Error: updating rollout revision that only happens in two spots in rollouts/sync.go the one on line 106 is a bit interesting to me, I will look into that one a bit more as well.

@alonbehaim Yea I missed that that is a good check to add to reduce noise.