kubectl: Kubectl returns pods in various states when only those in Running state are selected via --field-selector, or -o jsonpath

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): field-selector, field selector, status phase


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"b30876a5539f09684ff9fde266fda10b37738c9c", GitTreeState:"clean", BuildDate:"2018-01-16T21:52:38Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): Arch Linux
  • Kernel (e.g. uname -a): Linux something 4.16.3-1-ARCH #1 SMP PREEMPT Thu Apr 19 09:17:56 UTC 2018 x86_64 GNU/Linux
  • Install tools: ?
  • Others: ?

What happened:

kubectl get pods --field-selector=status.phase=Running --namespace some-ns

returns pods in other states than Running.

What you expected to happen: I expected only pods in state Running to be returned.

How to reproduce it (as minimally and precisely as possible):

kubectl get pods --field-selector=status.phase=Running --namespace some-ns

Anything else we need to know: That’s way too deep… I don’t think I can tackle that question here…

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 3
  • Comments: 64 (19 by maintainers)

Most upvoted comments

# Get all running pods in the namespace kubectl get pods --field-selector=status.phase=Running from https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-running-pods isn’t working as expected. Please reopen this issue.

Currently there’s no easy way to select running pods (as running means in English language) So please reopen this issue and do not close it until API supports this feature as it seems to be a basic feature to list running pods.

This is my dirty workaround for now: kubectl get pods -o jsonpath='{.items[*].status.containerStatuses[*].state.running},{.items[*].metadata.name}' --field-selector=status.phase==Running | sed 's/ /\n/' | grep startedAt | awk -F',' '{print $2}'

I’m sure you agree this is not how it’s supposed to be.

Actually, this is not a selector error, for some reason, a pod in a terminating state does not have a “Terminating” value in its yaml. doing the following command on a pod in terminating state kubectl get pod -l app=myapp -o yaml returns:

status: …phase: Running

By looking at the yaml, the only diffrenece between a running pod and a terminating pod
is that the terminating pod has the following 2 fields in metadata. It is not possible to use selectors to select/filter by these 2 fields

metadata: …deletionGracePeriodSeconds: 30 …deletionTimestamp:‘2018-12-11T15:42:14Z’

my workaround hack to getting running pods that are NOT terminating is with the following command :

kubectl get pod | grep Running

Get running pods and do not show Terminating pods (.metata.deletiion != nil)

kubectl get pods --field-selector=status.phase=Running --template {{range .items}}{{ if not .metadata.deletionTimestamp }}{{.metadata.name}}{{"\n"}}{{end}}{{end}}

I was curious to see if there was a way using only kubectl to get a list of the ready pods, and this is what I came up with (inspired by alahijani’s awk workaround above)

It uses a go-template to print the name of pods only if all of its containers are ready.

It’s definitely more complex than the awk workaround, but if you are looking for a pure kubectl way, or want to extend it with other conditions or fields, this go-template approach might be an option.

kubectl get pods -o=go-template --template='{{range .items}}{{$ready:=true}}{{range .status.containerStatuses}}{{if not .ready}}{{$ready = false}}{{end}}{{end}}{{if $ready}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}'

This did work for me though kubectl get pods --template '{{range .items}}{{ if not .metadata.deletionTimestamp }}{{.metadata.name}}{{"\n"}}{{end}}{{end}}'

I think this should be open again.

The docs clearly state

Get all running pods in the namespace $ kubectl get pods --field-selector=status.phase=Running

This command doesn’t get all pods in running state.

Call it a bug, a doc error, or even a UX issue of confusing command, but something seems missing here.

I should be able to query for running pods without doing some hack like kubectl get pods | grep Running

This still needs to be answered before we can proceed.

By “running pods” do you mean you do not want to see the ones that are still running, but in the process of terminating? Because that is what it is showing you now. When it says Terminating in the kubectl get output, that is technically still a pod that is in a Running status phase, it just shows you “Terminating” because it knows it is in the process of terminating, but that is not an actual status.

Showing the pod as Terminating is a convenience here. As Brian said the pod is indeed still in a state of Running but has a metadata.deletionTimestamp meaning the pod is marked for deletion.

Do you want the the convenience removed? This would show the same pods but with the state column of Running even if they are marked for deletion.

Do you want it to say Running (Terminating)?

@GMZwinge this is normal behavior. While termination is going, container has status Running, because it got termination signal and trying kindly complete the main thread. When it’s terminated pod changes his phase on state Terminated.
Here you may find answers on your questions.

I’ve simplified @brianpursley solution a bit.

Find NOT READY pods: kubectl -n default get pods -o=go-template --template='{{range $i := .items}}{{range .status.containerStatuses}}{{if not .ready}}{{$i.metadata.name}}{{"\n"}}{{end}}{{end}}{{end}}'

Find READY pods: kubectl -n default get pods -o=go-template --template='{{range $i := .items}}{{range .status.containerStatuses}}{{if .ready}}{{$i.metadata.name}}{{"\n"}}{{end}}{{end}}{{end}}'

@guai you can try https://gist.github.com/wknapik/315f52a20a2af43da35be5412625794a. I haven’t used it in a long time, so ymmv, but it used to work fine for me. There may be simpler/better ways, but I’ll leave those for someone else to cover ;]

I understand the issue with people wanting to get pods which are in terminating state, but at least there should be a flag to add to tell kubectl it really should only output pods which also show running state when calling kubectl get pods.

Thanks @DirkTheDaring, your solution outputs the desired information!

we have the same problem. kubectl get pods -n namespace -o jsonpath='{.items[*].status.containerStatuses[?(@.restartCount!=0)].image}' returns (images of) Pods that had multiple Restarts. I would like to do something like this with --filed-selector kubectl get pods -n namespace --watch=true --field-selector=status.containerStatuses.restartCount!=0 | ts '[%Y-%m-%d %H:%M:%.S]'

just using this: kubectl get pods -n namespace --watch=true | ts '[%Y-%m-%d %H:%M:%.S]' | grep -v " 0 "