argo-rollouts: Restarting 1-replica rollout causes stable to terminate immediately

Summary

In a Rollout with 1 replica, triggering a restart causes the running pod to be terminated immediately at the same time that the new pod starts up. This happens even when explicitly setting maxUnavailable: 0 (the default is documented as 25% which I would have expected to be rounded down to 0 pods). This causes no pods to be available until the new one becomes ready.

I know 1 replica is risky anyways but triggering a restart of a Deployment with 1 pod does what I would expect: it starts a new pod, waits for it to become ready, then terminates the old pod.

Diagnostics

Rollouts 1.1


Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 13
  • Comments: 17 (9 by maintainers)

Most upvoted comments

Hello, In my case, argo rollout restart causes random short downtime even with multiple replicas. It seems that traffic goes to killed pod for short time, which leads 500 error from ingress controller. (in my case AWS ingress controller). I guess kubectl.kubernetes.io/restartedAt feature might not support zero-downtime. (Stop forwarding traffic to pod, send kill signal to pod and wait for grace period, and kill. https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)

I hope that argo rollout restart follows normal rollout process like rollout restart does. Restarting pods for whatever reasons (eg. hygiene purposes, reloading secrets) SHOULD be tested like other change operations. Since there can be a lot of unexpected problems: cache stampede, updated secret be invalid value, etc. Safe rather than sorry.

Same here, how to make argo rollouts to restart pod one by one and not terminate old until new not become healthy

kubectl-argorollout-restart:

#!/usr/bin/env bash
set -euo pipefail

if [ "$#" -ne "2" ]; then
  echo "usage: $0 NAMESPACE ROLLOUT"
  exit 2
fi

NAMESPACE="$1"
ROLLOUT="$2"

paused="$(kubectl get rollout -ojsonpath="{.spec.paused}" -n "$NAMESPACE" "$ROLLOUT")"
if [ "$paused" = "true" ]; then
  echo "can't restart paused rollout, unpause by removing .spec.paused=true first" >&2
  exit 1
fi

kubectl patch rollout --type merge -n "$NAMESPACE" "$ROLLOUT" -p "$(cat <<EOF
{"spec": { "template": { "metadata": { "annotations": {
  "kubectl.kubernetes.io/restartedAt": "$(date --rfc-3339=seconds)"
}}}}}
EOF
)"