kubernetes: Flip pod's ready condition when node is unreachable
Getting hit by the new policy on skydns I shut down the host where it was running and waited for an automatic relocation. After a while (~1h) I manually stopped the pod and it was launched on a new one, as expected.
If host status is NotReady pods should be moved.
Here is the pod status when host was down :
{
"kind": "Pod",
"apiVersion": "v1beta3",
"metadata": {
"name": "kube-dns-787nc",
"generateName": "kube-dns-",
"namespace": "default",
"selfLink": "/api/v1beta1/pods/kube-dns-787nc?namespace=default",
"uid": "c32c0c6f-fb0a-11e4-83a8-42010af0b423",
"resourceVersion": "36141",
"creationTimestamp": "2015-05-15T14:00:37Z",
"labels": {
"k8s-app": "kube-dns",
"kubernetes.io/cluster-service": "true"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1beta3\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"kube-dns\",\"uid\":\"c3274113-fb0a-11e4-83a8-42010af0b423\",\"apiVersion\":\"v1beta3\",\"resourceVersion\":\"136\"}}"
}
},
"spec": {
"volumes": [
{
"name": "dns-token",
"hostPath": null,
"emptyDir": null,
"gcePersistentDisk": null,
"awsElasticBlockStore": null,
"gitRepo": null,
"secret": {
"secretName": "token-system-dns"
},
"nfs": null,
"iscsi": null,
"glusterfs": null
}
],
"containers": [
{
"name": "etcd",
"image": "gcr.io/google_containers/etcd:2.0.9",
"command": [
"/usr/local/bin/etcd",
"-listen-client-urls",
"http://127.0.0.1:2379,http://127.0.0.1:4001",
"-advertise-client-urls",
"http://127.0.0.1:2379,http://127.0.0.1:4001",
"-initial-cluster-token",
"skydns-etcd"
],
"resources": {},
"terminationMessagePath": "/dev/termination-log",
"imagePullPolicy": "IfNotPresent",
"capabilities": {}
},
{
"name": "kube2sky",
"image": "gcr.io/google_containers/kube2sky:1.4",
"args": [
"-domain=kubernetes.local",
"-kubecfg_file=/etc/dns_token/kubeconfig"
],
"resources": {},
"volumeMounts": [
{
"name": "dns-token",
"readOnly": true,
"mountPath": "/etc/dns_token"
}
],
"terminationMessagePath": "/dev/termination-log",
"imagePullPolicy": "IfNotPresent",
"capabilities": {}
},
{
"name": "skydns",
"image": "gcr.io/google_containers/skydns:2015-03-11-001",
"args": [
"-machines=http://localhost:4001",
"-addr=0.0.0.0:53",
"-domain=kubernetes.local."
],
"ports": [
{
"name": "dns",
"containerPort": 53,
"protocol": "UDP"
},
{
"name": "dns-tcp",
"containerPort": 53,
"protocol": "TCP"
}
],
"resources": {},
"livenessProbe": {
"exec": {
"command": [
"/bin/sh",
"-c",
"nslookup foobar 127.0.0.1 \u0026\u003e /dev/null; echo ok"
]
},
"initialDelaySeconds": 30,
"timeoutSeconds": 5
},
"terminationMessagePath": "/dev/termination-log",
"imagePullPolicy": "IfNotPresent",
"capabilities": {}
}
],
"restartPolicy": "Always",
"dnsPolicy": "Default",
"host": "internal-changecommerce-minion-006"
},
"status": {
"phase": "Running",
"Condition": [
{
"type": "Ready",
"status": "False"
}
],
"hostIP": "104.155.83.214",
"podIP": "10.244.6.2",
"containerStatuses": [
{
"name": "etcd",
"state": {
"running": {
"startedAt": "2015-05-15T18:25:16Z"
}
},
"lastState": {
"termination": {
"exitCode": 0,
"startedAt": "2015-05-15T15:12:48Z",
"finishedAt": "2015-05-15T18:23:57Z",
"containerID": "docker://1519a9ff8807f84ce55df761164460c66ba4a0e3f2eec0da7c650c9d121f4956"
}
},
"ready": true,
"restartCount": 1,
"image": "gcr.io/google_containers/etcd:2.0.9",
"imageID": "docker://b6b9a86dc06aa1361357ca1b105feba961f6a4145adca6c54e142c0be0fe87b0",
"containerID": "docker://43ab1b874434e97b868ed3e79437dd5dd0225a504b24e31a9490edb11f70cbfd"
},
{
"name": "kube2sky",
"state": {
"running": {
"startedAt": "2015-05-15T18:36:15Z"
}
},
"lastState": {
"termination": {
"exitCode": 255,
"startedAt": "2015-05-15T18:36:05Z",
"finishedAt": "2015-05-15T18:36:06Z",
"containerID": "docker://d58e3371d512551275314f45417a0ac0cb98897aa7296a7d132280e080e3b3d3"
}
},
"ready": false,
"restartCount": 15,
"image": "gcr.io/google_containers/kube2sky:1.4",
"imageID": "docker://3df83f1bccbf18ab59cd64da32cda26b4f56ae0cd6bb6e374477a60468f04232",
"containerID": "docker://919a5faa8a89dba0e975165995d32ba60b873d2b4e96a590fe840dee1b28a122"
},
{
"name": "skydns",
"state": {
"running": {
"startedAt": "2015-05-15T18:25:20Z"
}
},
"lastState": {
"termination": {
"exitCode": 2,
"startedAt": "2015-05-15T18:00:20Z",
"finishedAt": "2015-05-15T18:23:57Z",
"containerID": "docker://40a68db359a90b3aedcefbd960f51347841f21d65773140509dc045664b1a714"
}
},
"ready": true,
"restartCount": 9,
"image": "gcr.io/google_containers/skydns:2015-03-11-001",
"imageID": "docker://791ddf327076e0fd35a1125568a56c05ee1f1dfd7a165c74f4d489d8a5e65ac5",
"containerID": "docker://dd53be5827169136e55d768ac7d05a5c3664a2d1ba4c4b377b1d862f78993ca1"
}
]
}
}
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 31 (24 by maintainers)
Hi, i am a beginner in kubernetes. I met the similar question in my kubernetes cluster on the five vms.
v1.11.0Cluster have three masters and two nodes, when i shutdown master3 and two nodes, some pods status
NodeLost, but other pods stillRunning, Why?I describe one pod, Conditions Ready is
False, and service endpoint already delete this pod ip.kubectl get nodes