kubernetes: Flip pod's ready condition when node is unreachable

Getting hit by the new policy on skydns I shut down the host where it was running and waited for an automatic relocation. After a while (~1h) I manually stopped the pod and it was launched on a new one, as expected.

If host status is NotReady pods should be moved.

Here is the pod status when host was down :

{
    "kind": "Pod",
    "apiVersion": "v1beta3",
    "metadata": {
        "name": "kube-dns-787nc",
        "generateName": "kube-dns-",
        "namespace": "default",
        "selfLink": "/api/v1beta1/pods/kube-dns-787nc?namespace=default",
        "uid": "c32c0c6f-fb0a-11e4-83a8-42010af0b423",
        "resourceVersion": "36141",
        "creationTimestamp": "2015-05-15T14:00:37Z",
        "labels": {
            "k8s-app": "kube-dns",
            "kubernetes.io/cluster-service": "true"
        },
        "annotations": {
            "kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1beta3\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"kube-dns\",\"uid\":\"c3274113-fb0a-11e4-83a8-42010af0b423\",\"apiVersion\":\"v1beta3\",\"resourceVersion\":\"136\"}}"
        }
    },
    "spec": {
        "volumes": [
            {
                "name": "dns-token",
                "hostPath": null,
                "emptyDir": null,
                "gcePersistentDisk": null,
                "awsElasticBlockStore": null,
                "gitRepo": null,
                "secret": {
                    "secretName": "token-system-dns"
                },
                "nfs": null,
                "iscsi": null,
                "glusterfs": null
            }
        ],
        "containers": [
            {
                "name": "etcd",
                "image": "gcr.io/google_containers/etcd:2.0.9",
                "command": [
                    "/usr/local/bin/etcd",
                    "-listen-client-urls",
                    "http://127.0.0.1:2379,http://127.0.0.1:4001",
                    "-advertise-client-urls",
                    "http://127.0.0.1:2379,http://127.0.0.1:4001",
                    "-initial-cluster-token",
                    "skydns-etcd"
                ],
                "resources": {},
                "terminationMessagePath": "/dev/termination-log",
                "imagePullPolicy": "IfNotPresent",
                "capabilities": {}
            },
            {
                "name": "kube2sky",
                "image": "gcr.io/google_containers/kube2sky:1.4",
                "args": [
                    "-domain=kubernetes.local",
                    "-kubecfg_file=/etc/dns_token/kubeconfig"
                ],
                "resources": {},
                "volumeMounts": [
                    {
                        "name": "dns-token",
                        "readOnly": true,
                        "mountPath": "/etc/dns_token"
                    }
                ],
                "terminationMessagePath": "/dev/termination-log",
                "imagePullPolicy": "IfNotPresent",
                "capabilities": {}
            },
            {
                "name": "skydns",
                "image": "gcr.io/google_containers/skydns:2015-03-11-001",
                "args": [
                    "-machines=http://localhost:4001",
                    "-addr=0.0.0.0:53",
                    "-domain=kubernetes.local."
                ],
                "ports": [
                    {
                        "name": "dns",
                        "containerPort": 53,
                        "protocol": "UDP"
                    },
                    {
                        "name": "dns-tcp",
                        "containerPort": 53,
                        "protocol": "TCP"
                    }
                ],
                "resources": {},
                "livenessProbe": {
                    "exec": {
                        "command": [
                            "/bin/sh",
                            "-c",
                            "nslookup foobar 127.0.0.1 \u0026\u003e /dev/null; echo ok"
                        ]
                    },
                    "initialDelaySeconds": 30,
                    "timeoutSeconds": 5
                },
                "terminationMessagePath": "/dev/termination-log",
                "imagePullPolicy": "IfNotPresent",
                "capabilities": {}
            }
        ],
        "restartPolicy": "Always",
        "dnsPolicy": "Default",
        "host": "internal-changecommerce-minion-006"
    },
    "status": {
        "phase": "Running",
        "Condition": [
            {
                "type": "Ready",
                "status": "False"
            }
        ],
        "hostIP": "104.155.83.214",
        "podIP": "10.244.6.2",
        "containerStatuses": [
            {
                "name": "etcd",
                "state": {
                    "running": {
                        "startedAt": "2015-05-15T18:25:16Z"
                    }
                },
                "lastState": {
                    "termination": {
                        "exitCode": 0,
                        "startedAt": "2015-05-15T15:12:48Z",
                        "finishedAt": "2015-05-15T18:23:57Z",
                        "containerID": "docker://1519a9ff8807f84ce55df761164460c66ba4a0e3f2eec0da7c650c9d121f4956"
                    }
                },
                "ready": true,
                "restartCount": 1,
                "image": "gcr.io/google_containers/etcd:2.0.9",
                "imageID": "docker://b6b9a86dc06aa1361357ca1b105feba961f6a4145adca6c54e142c0be0fe87b0",
                "containerID": "docker://43ab1b874434e97b868ed3e79437dd5dd0225a504b24e31a9490edb11f70cbfd"
            },
            {
                "name": "kube2sky",
                "state": {
                    "running": {
                        "startedAt": "2015-05-15T18:36:15Z"
                    }
                },
                "lastState": {
                    "termination": {
                        "exitCode": 255,
                        "startedAt": "2015-05-15T18:36:05Z",
                        "finishedAt": "2015-05-15T18:36:06Z",
                        "containerID": "docker://d58e3371d512551275314f45417a0ac0cb98897aa7296a7d132280e080e3b3d3"
                    }
                },
                "ready": false,
                "restartCount": 15,
                "image": "gcr.io/google_containers/kube2sky:1.4",
                "imageID": "docker://3df83f1bccbf18ab59cd64da32cda26b4f56ae0cd6bb6e374477a60468f04232",
                "containerID": "docker://919a5faa8a89dba0e975165995d32ba60b873d2b4e96a590fe840dee1b28a122"
            },
            {
                "name": "skydns",
                "state": {
                    "running": {
                        "startedAt": "2015-05-15T18:25:20Z"
                    }
                },
                "lastState": {
                    "termination": {
                        "exitCode": 2,
                        "startedAt": "2015-05-15T18:00:20Z",
                        "finishedAt": "2015-05-15T18:23:57Z",
                        "containerID": "docker://40a68db359a90b3aedcefbd960f51347841f21d65773140509dc045664b1a714"
                    }
                },
                "ready": true,
                "restartCount": 9,
                "image": "gcr.io/google_containers/skydns:2015-03-11-001",
                "imageID": "docker://791ddf327076e0fd35a1125568a56c05ee1f1dfd7a165c74f4d489d8a5e65ac5",
                "containerID": "docker://dd53be5827169136e55d768ac7d05a5c3664a2d1ba4c4b377b1d862f78993ca1"
            }
        ]
    }
}

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 31 (24 by maintainers)

Most upvoted comments

Hi, i am a beginner in kubernetes. I met the similar question in my kubernetes cluster on the five vms.

  • Kubernetes version: v1.11.0
  • OS: CentOS Linux release 7.5.1804 (Core)

Cluster have three masters and two nodes, when i shutdown master3 and two nodes, some pods status NodeLost, but other pods still Running, Why?

I describe one pod, Conditions Ready is False, and service endpoint already delete this pod ip.

kubectl get nodes

NAME                 STATUS     ROLES     AGE       VERSION
master1              Ready      master    6d        v1.11.0
master2              Ready      master    6d        v1.11.0
master3              NotReady   master    6d        v1.11.0
node.9cb7            NotReady   <none>    21h       v1.11.0
node1                NotReady   <none>    6d        v1.11.0