kubernetes: Job being constanly recreated despite RestartPolicy: Never

For any future readers, setting restartPolicy: OnFailure will prevent the neverending creation of pods because it will just restart the failing one.

If you want to create new pods on failure with restartPolicy: Never, you can limit them by setting activeDeadlineSeconds – Upon reaching the deadline without success, the job will have status with reason: DeadlineExceeded. No more pods will be created, and existing pods will be deleted.

Trying to parse “Generally speaking the restart policy is applied to a pod not to a job, as you might understood initially. k8s role is to actually run a pod to successful completion, it cannot judge based on the exit code whether that failure was expected and should re-run or it was not and should stop re-trying. It’s user job to do that distinction based on the output from k8s.” @soltysh

Does restartPolicy of a job have any meaning whatsoever? If jobs communicate their failure as an exit code, then how can the scheduler make the conclusion about what has failed? If jobs do not communicate their result as an exit code then how are they expected to do that instead?

In essence, i would like to trigger a job once, communicate failure/success somehow (i do this with an exit code now), and inspect the log to see what might have gone wrong/right.

Quite a bit late, but if anyone else looks for it’s https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy

@soltysh: the biggest problem with ‘try to reach completions’ that it create many dead/error pods objects in etcd which significantly slow down all api calls (and of course delete take forever). It looks like use need to implement watchdog for all jobs ):

@innovia .spec.backoffLimit introduced in https://github.com/kubernetes/kubernetes/pull/48075 and https://github.com/kubernetes/kubernetes/pull/51153 is what you’ll be looking for.

I was still confused on why restartPolicy: Never restarts the job and I wanted to know the difference between OnFailure and Never. I found openshift documentation page which helped me understand better. Hopefully this helps someone.

The Job specification restart policy only applies to the pods, and not the job controller. However, the job controller is hard-coded to keep retrying Jobs to completion.

As such, restartPolicy: Never or --restart=Never results in the same behavior as restartPolicy: OnFailure or --restart=OnFailure. That is, when a Job fails it is restarted automatically until it succeeds (or is manually discarded). The policy only sets which subsystem performs the restart.

With the Never policy, the job controller performs the restart. With each attempt, the job controller increments the number of failures in the Job status and create new pods. This means that with each failed attempt, the number of pods increases.

With the OnFailure policy, kubelet performs the restart. Each attempt does not increment the number of failures in the Job status. In addition, kubelet will retry failed Jobs starting pods on the same nodes.

Lastly, as kubernetes documentation states when in doubt use Never

Sounds like this is resolved.

Also, StackOverflow is the best place to ask questions like this, since it makes the answers much more findable by people who have similar questions in the future.

This is by design, see here. Generally speaking the restart policy is applied to a pod not to a job, as you might understood initially. k8s role is to actually run a pod to successful completion, it cannot judge based on the exit code whether that failure was expected and should re-run or it was not and should stop re-trying. It’s user job to do that distinction based on the output from k8s.

@soltysh: I appreciate your effort, but still not convinced that I need to write my own controller just for restart (of course we can use pod instead of job, but it will be nice to have consistent api/behavior between job and scheduled jobs). Any thoughts about adding restartPolicy at Job level?