kubernetes: kubectl shows Pod status Completed if one of the containers exited with an Error

What happened:

Running a Job with two or more containers creates a Pod. If one of the container fails such Pod status in kubectl is shown as Completed if the name of such container is the last name of the container names ordered alphabetically.

What you expected to happen:

If the container who’s name is the last name of the container names ordered alphabetically fails the Pod status in kubectl should be shown as Error.

How to reproduce it (as minimally and precisely as possible):

cat << EOF | kubectl create -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: test
spec:
  backoffLimit: 0
  completionMode: NonIndexed
  parallelism: 1
  suspend: false
  template:
    spec:
      restartPolicy: Never
      containers:
      - command:
        - /bin/sh
        - -exc
        - "true"
        name: a
        image: busybox
      - command:
        - /bin/sh
        - -exc
        - "false"
        name: z
        image: busybox
EOF

Anything else we need to know?:

Environment:

Kubernetes client and server versions (use kubectl version):

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.22.17

Cloud provider or hardware configuration:

kind v0.15.0 go1.19 linux/amd64

OS (e.g: cat /etc/os-release):

$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

About this issue

Original URL
State: open
Created 8 months ago
Comments: 15 (9 by maintainers)

Most upvoted comments

[root@qa-vpc-cni-node-3-s57fhiyhso4f ~]# kubectl get job
NAME   COMPLETIONS   DURATION   AGE
test   0/1           65s        65s
[root@qa-vpc-cni-node-3-s57fhiyhso4f ~]# kubectl describe job test
Name:             test
Namespace:        default
Selector:         batch.kubernetes.io/controller-uid=8c391b14-81d9-448e-8460-052b0ef5eb71
Labels:           batch.kubernetes.io/controller-uid=8c391b14-81d9-448e-8460-052b0ef5eb71
                  batch.kubernetes.io/job-name=test
                  controller-uid=8c391b14-81d9-448e-8460-052b0ef5eb71
                  job-name=test
Annotations:      batch.kubernetes.io/job-tracking: 
Parallelism:      1
Completions:      <unset>
Completion Mode:  NonIndexed
Start Time:       Thu, 02 Nov 2023 21:22:02 +0800
Pods Statuses:    0 Active (0 Ready) / 0 Succeeded / 1 Failed
Pod Template:
  Labels:  batch.kubernetes.io/controller-uid=8c391b14-81d9-448e-8460-052b0ef5eb71
           batch.kubernetes.io/job-name=test
           controller-uid=8c391b14-81d9-448e-8460-052b0ef5eb71
           job-name=test
  Containers:
   a:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -exc
      true
    Environment:  <none>
    Mounts:       <none>
   z:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -exc
      false
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type     Reason                Age   From            Message
  ----     ------                ----  ----            -------
  Normal   SuccessfulCreate      80s   job-controller  Created pod: test-2hgpg
  Warning  BackoffLimitExceeded  73s   job-controller  Job has reached the specified backoff limit

kube version

[root@qa-vpc-cni-node-3-s57fhiyhso4f ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.6-es", GitCommit:"aa2c39c6a6c61456ba2bfe7dc466ed2e4249a3a6", GitTreeState:"clean", BuildDate:"2023-09-15T09:23:11Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.6-es", GitCommit:"aa2c39c6a6c61456ba2bfe7dc466ed2e4249a3a6", GitTreeState:"clean", BuildDate:"2023-09-15T09:22:46Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

no recurrence in v1.27

[root@node-4 ~]# kubectl get job  test
NAME   COMPLETIONS   DURATION   AGE
test   0/1           38s        38s
[root@node-4 ~]# kubectl describe  job  test
Name:             test
Namespace:        default
Selector:         batch.kubernetes.io/controller-uid=a314c54c-70af-4eb2-8842-eaa6975bd443
Labels:           batch.kubernetes.io/controller-uid=a314c54c-70af-4eb2-8842-eaa6975bd443
                  batch.kubernetes.io/job-name=test
                  controller-uid=a314c54c-70af-4eb2-8842-eaa6975bd443
                  job-name=test
Annotations:      <none>
Parallelism:      1
Completions:      <unset>
Completion Mode:  NonIndexed
Start Time:       Thu, 02 Nov 2023 21:31:42 +0800
Pods Statuses:    0 Active (0 Ready) / 0 Succeeded / 1 Failed
Pod Template:
  Labels:  batch.kubernetes.io/controller-uid=a314c54c-70af-4eb2-8842-eaa6975bd443
           batch.kubernetes.io/job-name=test
           controller-uid=a314c54c-70af-4eb2-8842-eaa6975bd443
           job-name=test
  Containers:
   a:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -exc
      true
    Environment:  <none>
    Mounts:       <none>
   z:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -exc
      false
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type     Reason                Age   From            Message
  ----     ------                ----  ----            -------
  Normal   SuccessfulCreate      45s   job-controller  Created pod: test-kc6kh
  Warning  BackoffLimitExceeded  28s   job-controller  Job has reached the specified backoff limit
[root@node-4 ~]# kubectl version
Client Version: v1.28.2-es
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2-es

No recurrence in v1.28.2 Maybe we need more information and other reproduction methods or does Kubernetes v1.28.0 has this bug?

Goend on Nov 2, 2023