argo-rollouts: Restarting 1-replica rollout causes stable to terminate immediately
Summary
In a Rollout with 1 replica, triggering a restart causes the running pod to be terminated immediately at the same time that the new pod starts up. This happens even when explicitly setting maxUnavailable: 0 (the default is documented as 25% which I would have expected to be rounded down to 0 pods). This causes no pods to be available until the new one becomes ready.
I know 1 replica is risky anyways but triggering a restart of a Deployment with 1 pod does what I would expect: it starts a new pod, waits for it to become ready, then terminates the old pod.
Diagnostics
Rollouts 1.1
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 13
- Comments: 17 (9 by maintainers)
Hello, In my case,
argo rollout restartcauses random short downtime even with multiple replicas. It seems that traffic goes to killed pod for short time, which leads 500 error from ingress controller. (in my case AWS ingress controller). I guesskubectl.kubernetes.io/restartedAtfeature might not support zero-downtime. (Stop forwarding traffic to pod, send kill signal to pod and wait for grace period, and kill. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)I hope that
argo rollout restartfollows normal rollout process likerollout restartdoes. Restarting pods for whatever reasons (eg. hygiene purposes, reloading secrets) SHOULD be tested like other change operations. Since there can be a lot of unexpected problems: cache stampede, updated secret be invalid value, etc. Safe rather than sorry.Same here, how to make argo rollouts to restart pod one by one and not terminate old until new not become healthy
kubectl-argorollout-restart: