kubernetes: Kubectl reports wrong status for evicted pod when container statuses are synced after

What happened:

In Kubernetes <1.22, Evicted pods would not have any container status synced. Kubectl displayed them as “Evicted”.

In 1.22+, we fixed a bug where we now sync out the last known container status from the runtime on Evicted pods. However, that status syncs after we update the pod status. As soon as the runtime status syncs, since it is more recent, it takes precedence in the display, and the pod’s status updates from Evicted to whatever the container’s status is (Error, ContainerStatusUnknown, etc.)

What you expected to happen:

So long as the pod status stays Evicted with phase Failed, I expected kubectl to print Evicted.

How to reproduce it (as minimally and precisely as possible):

See https://github.com/kubernetes/kubernetes/issues/105358#issue-1011532980 and https://github.com/kubernetes/kubernetes/issues/105358#issuecomment-932470027

Anything else we need to know?:

/sig cli node

Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 3
  • Comments: 15 (8 by maintainers)

Most upvoted comments

@ehashman The root cause of the kubectl display issue is that we have changed that previously we never reported container status after an eviction or preemption and so on, Now we sync out the last known container status from the runtime even the pod has been evicted, preempted or teminated by node shutdown. so if we keep the current status report logic, we should change the kubectl printer in https://github.com/kubernetes/kubernetes/blob/0cef26182cf83091317c17ebb28918391ecf3583/pkg/printers/internalversion/printers.go#L741, we may change the common display to special for eviction or preemption and so on, is this reasonable?