kubernetes: CrashLoopBackOff and confusing status

Pods are crashing as reported by:

$ kubectl get po   
NAME                  READY     STATUS             RESTARTS   AGE
elasticsearch-attiy   0/1       CrashLoopBackOff   25         1h
elasticsearch-lkl4i   0/1       CrashLoopBackOff   25         1h
frontend-81lqv        1/1       Running            0          1h
frontend-ddanc        1/1       Running            0          1h

But the message isn’t clear when trying to exec:

$ kubectl exec -it elasticsearch-attiy sh 
error: Error executing remote command: Error executing command in container: container not found ("elasticsearch")

And the describe gives an even more confusion “2/2 replicas created” and it did find the image:

$ kubectl describe po elastic
search-attiy
Name:                           elasticsearch-attiy
Namespace:                      default
Image(s):                       eu.gcr.io/foo/foo_elasticsearch:latest
Node:                           gke-small-cluster-7834f89-node-wke5/10.210.0.1
Labels:                         name=elasticsearch
Status:                         Running
Reason:
Message:
IP:                             10.180.1.11
Replication Controllers:        elasticsearch (2/2 replicas created)
Containers:
  elasticsearch:
    Image:              eu.gcr.io/foo/foo_elasticsearch:latest
    State:              Waiting
      Reason:           CrashLoopBackOff
    Ready:              False
    Restart Count:      26
Conditions:
  Type          Status
  Ready         False 
Events:
  FirstSeen                             LastSeen                        Count   From                                 S
...
  Wed, 18 Nov 2015 21:24:42 +0100       Wed, 18 Nov 2015 23:09:03 +0100 26      {kubelet gke-small-cluster-7834f89-node-wke5}        spec.containers{elasticsearch}  Pulled  Successfully pulled image "eu.gcr.io/foo/foo_elasticsearch:latest"
  Wed, 18 Nov 2015 23:09:03 +0100       Wed, 18 Nov 2015 23:09:03 +0100 1       {kubelet gke-small-cluster-7834f89-node-wke5}        spec.containers{elasticsearch}  Created Created with docker id a7429f37f133
  Wed, 18 Nov 2015 23:09:04 +0100       Wed, 18 Nov 2015 23:09:04 +0100 1       {kubelet gke-small-cluster-7834f89-node-wke5}        spec.containers{elasticsearch}  Started Started with docker id a7429f37f133

It would help to have clearer messages to help understand causes of crashes.

The exec command could say that the given container isn’t running properly on the Pod (with a status or a command that could describe more in details the reasons).

The describe command here could clarify that replicas have been created but aren’t healthy. More importantly I found missing the reason of why the container got restarted. I do suspect that my command statement is the cause.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Reactions: 14
  • Comments: 15 (7 by maintainers)

Most upvoted comments

The actually useful command to execute would have been:

$ kubectl logs elasticsearch-attiy -p

I was having this problem, until I changed the FROM line in my Dockerfile from:

FROM microsoft/aspnetcore:2.0.0 … to … FROM microsoft/aspnetcore:2.0.5

(I’m running in a linux docker image. Not sure what it would be if windows)