kubernetes: activeDeadlineSeconds not working correctly for Jobs in 1.24.0

What happened?

Prior to 1.24.0 setting spec.adctiveDeadlineSeconds in a Job object (batch/v1) would kill the Job and Pods if they ran longer than the integer specified. I tested today with 1.24.0 and it’s not working. The Job defined below (for testing) should be killed 10 seconds, but isn’t. It’s actually being terminated after about 55 seconds.

apiVersion: batch/v1
kind: Job
metadata:
  name: test
spec:
  activeDeadlineSeconds: 10
  ttlSecondsAfterFinished: 120
  completions: 5
  parallelism: 1
  backoffLimit: 4
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: ctr
        image: alpine:latest
        command: ['sh', '-c', 'echo "TEST" && sleep 60']

What did you expect to happen?

The Job to be killed after 10 seconds as per activeDeadlineSeconds: 10. It’s worked like this in 1.23.x.

How can we reproduce it (as minimally and precisely as possible)?

On a cluster running 1.24.0 with the following YAML file called job.yml.

job.yml

apiVersion: batch/v1
kind: Job
metadata:
  name: test
spec:
  activeDeadlineSeconds: 10
  ttlSecondsAfterFinished: 120
  completions: 5
  parallelism: 1
  backoffLimit: 4
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: ctr
        image: alpine:latest
        command: ['sh', '-c', 'echo "TEST" && sleep 60']

Create the Job. kubectl apply -f job.yml

On Kubernetes clusters earlier than 1.24.0 it would fail after 10 seconds and the following command should show Warning DeadlineExceeded Xs job-controller Job was active longer than specified deadline in the events 10 seconds after the Pod is created.

kubectl describe job test

<SNIP>
Events:
  Type     Reason            Age   From            Message
  ----     ------            ----  ----            -------
  Normal   SuccessfulCreate  16s   job-controller  Created pod: ckad1-kk6jt
  Normal   SuccessfulDelete  6s    job-controller  Deleted pod: ckad1-kk6jt
  Warning  DeadlineExceeded  6s    job-controller  Job was active longer than specified deadline

In K8s 1.24.0 it’s terminated after about a minute instead of 10 seconds. It works as expected in 1.23.6 and other point versions of 1.23.

Anything else we need to know?

It works as expected, and documented, in all versions of 1.23 I’ve tested on.

Docs: https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/arm64"}

Cloud provider

OS version

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, …) and versions (if applicable)