kubernetes: kube-apiserver fails to start

Looking in kubelet’s logs, I see: “API error (500): Cannot start container <ID>: [8] System error: read parent: connection reset by peer”

I have only been able to consistently reproduce it on a dev build of kubernetes running in GKE, but I think I saw the same thing happen off and on when debugging some of the Jenkins GKE-Test failures from Friday.

More logs:

event.go:203] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67", UID:"10230b2ecdd307213211dedb5c7bd1c4", APIVersion:"v1", R
esourceVersion:"", FieldPath:"spec.containers{kube-apiserver}"}): reason: 'created' Created with docker id a76faba71e1615d5a638590c6c0d8e303f93d8c0c1702e7af4b42c84d59d210e
E0615 21:53:54.719591    2464 manager.go:1491] Error running pod "kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67_default" container "kube-apiserver": API error (500): Cannot start container a76faba71e1615d5a638590c6c
0d8e303f93d8c0c1702e7af4b42c84d59d210e: [8] System error: read parent: connection reset by peer
I0615 21:53:54.719648    2464 event.go:203] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67", UID:"10230b2ecdd307213211dedb5c7bd1c4", APIVersion:"v1", R
esourceVersion:"", FieldPath:"spec.containers{kube-apiserver}"}): reason: 'failed' Failed to start with docker id a76faba71e1615d5a638590c6c0d8e303f93d8c0c1702e7af4b42c84d59d210e with error: API error (500): Cannot start c
ontainer a76faba71e1615d5a638590c6c0d8e303f93d8c0c1702e7af4b42c84d59d210e: [8] System error: read parent: connection reset by peer
W0615 21:53:54.720978    2464 container.go:249] Failed to create summary reader for "/a76faba71e1615d5a638590c6c0d8e303f93d8c0c1702e7af4b42c84d59d210e": none of the resources are being tracked.
E0615 21:54:04.427496    2464 kubelet.go:1574] error getting node: node gke-664608984929-3a3afa5a2b3942c0cf67 not found
I0615 21:54:04.577893    2464 event.go:203] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67", UID:"10230b2ecdd307213211dedb5c7bd1c4", APIVersion:"v1", R
esourceVersion:"", FieldPath:"spec.containers{kube-apiserver}"}): reason: 'created' Created with docker id 6a170b2cf16b426df4cf0b4631cafeef6f7dddbb9c71624960ac9b4f4a2f2e8b
E0615 21:54:04.769703    2464 manager.go:1491] Error running pod "kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67_default" container "kube-apiserver": API error (500): Cannot start container 6a170b2cf16b426df4cf0b4631
cafeef6f7dddbb9c71624960ac9b4f4a2f2e8b: [8] System error: read parent: connection reset by peer
I0615 21:54:04.769741    2464 event.go:203] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67", UID:"10230b2ecdd307213211dedb5c7bd1c4", APIVersion:"v1", R
esourceVersion:"", FieldPath:"spec.containers{kube-apiserver}"}): reason: 'failed' Failed to start with docker id 6a170b2cf16b426df4cf0b4631cafeef6f7dddbb9c71624960ac9b4f4a2f2e8b with error: API error (500): Cannot start c
ontainer 6a170b2cf16b426df4cf0b4631cafeef6f7dddbb9c71624960ac9b4f4a2f2e8b: [8] System error: read parent: connection reset by peer
W0615 21:54:04.771218    2464 container.go:249] Failed to create summary reader for "/6a170b2cf16b426df4cf0b4631cafeef6f7dddbb9c71624960ac9b4f4a2f2e8b": none of the resources are being tracked.
E0615 21:54:14.434616    2464 kubelet.go:1574] error getting node: node gke-664608984929-3a3afa5a2b3942c0cf67 not found
I0615 21:54:14.615883    2464 event.go:203] Event(api.ObjectReference{Kind:"Pod", Namespace:"default", Name:"kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67", UID:"10230b2ecdd307213211dedb5c7bd1c4", APIVersion:"v1", R
esourceVersion:"", FieldPath:"spec.containers{kube-apiserver}"}): reason: 'created' Created with docker id 3e0201804da0616ed9942635b54d9b28c804e2ca590569f304ea3a6ea5d10442
E0615 21:54:14.800110    2464 manager.go:1491] Error running pod "kube-apiserver-gke-664608984929-3a3afa5a2b3942c0cf67_default" container "kube-apiserver": API error (500): Cannot start container 3e0201804da0616ed9942635b5
4d9b28c804e2ca590569f304ea3a6ea5d10442: [8] System error: read parent: connection reset by peer