origin: Pods stuck in Terminating in 3.2.0.5

@metal3d oc delete pod/<name of pod> --grace-period=0 will force deletion.

@anandbaskaran if command provided by @ncdc still hangs, you can try forcing it with --force. It gives: oc delete pod/<name of pod> --grace-period=0 --force

Upstream PR: https://github.com/kubernetes/kubernetes/pull/23746

I was finally able to reproduce this on a 3 node cluster after many hours of creating 500 pods, waiting for 200 to run, and then tearing down the project. For many hours, it was fine, but I did notice nodes were getting less and less successful at getting the full set of 200 pods in a running state as test ran more and more, but it was always able to properly tear-down all the pods.

After much time, I was left in a state where finally 1 of the nodes failed to tear down 3 pods. The node continued to report a valid heartbeat back to the API server, but it would no longer launch new pods that were scheduled to it. I was able to ssh into the machine and do a little more sleuthing.

The kubelet actually did see the notification from the watch source about the pod:

I0331 20:15:30.501759    3952 kubelet.go:2394] SyncLoop (ADD, "api"): "test-034xr_e2e-tests-nodestress-8e5yg(f03e60df-f77c-11e5-bed9-080027242396)"
I0331 20:15:30.823634    3952 manager.go:1688] Need to restart pod infra container for "test-034xr_e2e-tests-nodestress-8e5yg(f03e60df-f77c-11e5-bed9-080027242396)" because it is not found
I0331 20:16:59.570316    3952 kubelet.go:2420] SyncLoop (PLEG): "test-034xr_e2e-tests-nodestress-8e5yg(f03e60df-f77c-11e5-bed9-080027242396)", event: &pleg.PodLifecycleEvent{ID:"f03e60df-f77c-11e5-bed9-080027242396", Type:"ContainerStarted", Data:"4749d3890056803ec6d599cd82391d459d8d47246bead00764cb590f2ea2991f"}
I0331 20:23:20.586986    3952 kubelet.go:2401] SyncLoop (UPDATE, "api"): "test-034xr_e2e-tests-nodestress-8e5yg(f03e60df-f77c-11e5-bed9-080027242396)"

Looking at the container in question:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicationController","namespace":"e2e-tests-nodestress-8e5yg","name":"test","uid":"ededf40d-f77c-11e5-bed9-080027242396","apiVersion":"v1","resourceVersion":"18868"}}
  creationTimestamp: 2016-03-31T20:12:48Z
  deletionGracePeriodSeconds: 30
  deletionTimestamp: 2016-03-31T20:23:50Z
  generateName: test-
  labels:
    name: test
  name: test-034xr
  namespace: e2e-tests-nodestress-8e5yg
  resourceVersion: "19910"
  selfLink: /api/v1/namespaces/e2e-tests-nodestress-8e5yg/pods/test-034xr
  uid: f03e60df-f77c-11e5-bed9-080027242396
spec:
  containers:
  - image: openshift/hello-openshift
    imagePullPolicy: Always
    name: test
    resources: {}
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-4s8wq
      readOnly: true
  dnsPolicy: ClusterFirst
  nodeName: kubernetes-node-1
  restartPolicy: Always
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - name: default-token-4s8wq
    secret:
      secretName: default-token-4s8wq
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2016-03-31T20:15:30Z
    message: 'containers with unready status: [test]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  containerStatuses:
  - image: openshift/hello-openshift
    imageID: ""
    lastState: {}
    name: test
    ready: false
    restartCount: 0
    state:
      waiting:
        message: 'Image: openshift/hello-openshift is ready, container is creating'
        reason: ContainerCreating
  hostIP: 10.245.1.3
  phase: Pending
  startTime: 2016-03-31T20:15:30Z

You can see it was stuck in waiting state with container creating status.

Name:               test-034xr
Namespace:          e2e-tests-nodestress-8e5yg
Node:               kubernetes-node-1/10.245.1.3
Start Time:         Thu, 31 Mar 2016 16:15:30 -0400
Labels:             name=test
Status:             Terminating (expires Thu, 31 Mar 2016 16:23:50 -0400)
Termination Grace Period:   30s
IP:             
Controllers:            ReplicationController/test
Containers:
  test:
    Container ID:   
    Image:      openshift/hello-openshift
    Image ID:       
    Port:       
    QoS Tier:
      cpu:          BestEffort
      memory:           BestEffort
    State:          Waiting
      Reason:           ContainerCreating
    Ready:          False
    Restart Count:      0
    Environment Variables:  <none>
Conditions:
  Type      Status
  Ready     False 
Volumes:
  default-token-4s8wq:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-4s8wq
No events.

Looking at the docker logs:

time="2016-03-31T23:09:46.782184154Z" level=info msg="{Action=create, LoginUID=4294967295, PID=3952}"
time="2016-03-31T23:10:01.782790028Z" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

So my current theory is the following:

1. we are subject to this flake if a container is in creating state
2. docker is doing something unexpected

I have stashed the logs away to analyze more tomorrow.