kubernetes: Corrupted/bitflipped serialized API data present in etcd

/kind bug

What happened: API server became unable to respond to requests for pod resources. There is an error in the logs deserializing the Pod protobuf.

If we look at etcd we see two entries for a datadog-agent pod ending in dxjt9, where there should only be one (the ‘n’ in ‘agent’ changes to ‘l’):

• datadog-agent-dxjt9
• datadog-agelt-dxjt9

Looking at the data for these two records in etcd we can see multiple places where the data differs between them, including the entity’s name and the type for WorkingDir being changed from 2 (length-delimited) to 0 (varint) which is the error the API server logs are showing.

There is no call to the API server for a resource named datadog-agelt-dxjt9. It seems that the key and data is corrupted somewhere between kube-apiserver and etcd or in etcd itself.

etcd-records.zip

A binary diff of the two records shows multiple bits flipped. There are a few interesting things about these flipped bits.

• It is always the ‘2’ bit in a byte.
• After the first flipped bit, they are all at byte offset ending in 0.
• The first is in the etcd key (agent -> agelt).
• The others are in the serialized podspec data
• They changed values alternate between +2 and -2.

offset: 0x22 (34), good: 6e bad: 6c 
offset: 0xa0 (160), good: d9 bad: db 
offset: 0x120 (288), good: 7 bad: 5 
offset: 0x220 (544), good: 61 bad: 63 
offset: 0x260 (608), good: 2a bad: 28 
offset: 0x4e0 (1248), good: 2d bad: 2f 
offset: 0x760 (1888), good: 2e bad: 2c 
offset: 0x7a0 (1952), good: 5 bad: 7 

Manually removing the corrupted record with etcdctl returned the cluster to a working state.

API server logs: Host 1:

I0722 02:14:16.832788       1 wrap.go:42] PUT /api/v1/namespaces/default/pods/datadog-agent-dxjt9/status: (3.293948ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
I0722 02:14:41.932049       1 wrap.go:42] GET /api/v1/namespaces/default/pods/datadog-agent-dxjt9: (2.369567ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
I0722 02:14:41.936873       1 wrap.go:42] PUT /api/v1/namespaces/default/pods/datadog-agent-dxjt9/status: (3.532971ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
I0722 02:15:07.038952       1 wrap.go:42] GET /api/v1/namespaces/default/pods/datadog-agent-dxjt9: (2.708027ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
I0722 02:15:07.046041       1 wrap.go:42] PUT /api/v1/namespaces/default/pods/datadog-agent-dxjt9/status: (5.724827ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
I0722 02:16:04.449918       1 wrap.go:42] PATCH /api/v1/namespaces/default/events/datadog-agent-dxjt9.153cbf96727afa55: (9.020874ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
...
I0722 02:20:10.061938       1 wrap.go:42] GET /api/v1/namespaces/default/pods/datadog-agent-dxjt9: (3.513629ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
E0722 02:20:10.068328       1 watcher.go:268] failed to prepare current and previous objects: proto: wrong wireType = 0 for field WorkingDir
W0722 02:20:10.068400       1 reflector.go:341] storage/cacher.go:/pods: watch of *core.Pod ended with: Internal error occurred: proto: wrong wireType = 0 for field WorkingDir
I0722 02:20:10.068754       1 wrap.go:42] PUT /api/v1/namespaces/default/pods/datadog-agent-dxjt9/status: (5.488655ms) 200 [[kubelet/v1.10.3 (linux/amd64) kubernetes/2bba012] XX.XX.154.151:3002]
...
I0722 02:20:11.071738       1 get.go:238] Starting watch for /api/v1/pods, rv=4018844 labels= fields= timeout=8m18s
E0722 02:20:11.072657       1 cacher.go:271] unexpected ListAndWatch error: storage/cacher.go:/pods: Failed to list *core.Pod: proto: wrong wireType = 0 for field WorkingDir
I0722 02:20:11.074093       1 wrap.go:42] GET /api/v1/namespaces/kube-system/secrets/aws-node-token-9rwj8: (2.031495ms) 200 [[kube-apiserver/v1.10.3 (linux/amd64) kubernetes/2bba012] 127.0.0.1:34804]
I0722 02:20:11.078168       1 wrap.go:42] GET /api/v1/namespaces/kube-system/serviceaccounts/aws-node: (3.485772ms) 200 [[kube-apiserver/v1.10.3 (linux/amd64) kubernetes/2bba012] 127.0.0.1:34804]
I0722 02:20:11.078760       1 get.go:238] Starting watch for /api/v1/pods, rv=4018844 labels= fields= timeout=7m53s
...
E0722 02:22:31.220597       1 status.go:64] apiserver received an error that is not an metav1.Status: rpc error: code = Unknown desc = proto: wrong wireType = 0 for field Key
I0722 02:22:31.220825       1 wrap.go:42] PUT /api/v1/namespaces/ticketing/pods/uts-queuemanager-production-df64b6d54-gq5qv/status: (207.160954ms) 500
goroutine 240059803 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc432e987e0, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc432e987e0, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc42336cde0, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc42f13e2a0, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x3d863b2, 0x23, 0x7fa4a5f62440, 0xc428a2d780, 0x9868e80, 0xc42e1e0448, 0xc422e80500, 0x1f4, 0x984fc40, 0xc42fb29200)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:95 +0x8d
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fa4a5afce20, 0xc42f0ade00, 0x986b940, 0xc420a75ce0, 0x0, 0x0, 0x3d209f1, 0x2, 0x9868e80, 0xc42e1e0448, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:123 +0x32b
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fa4a5afce20, 0xc42f0ade00, 0x983ce00, 0xc42f0add10, 0x986b940, 0xc420a75ce0, 0x0, 0x0, 0x3d209f1, 0x2, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:142 +0x163
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc428c10000, 0x983ce00, 0xc42f0add10, 0x9868e80, 0xc42e1e0448, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.UpdateResource.func1(0x9868e80, 0xc42e1e0448, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/update.go:112 +0x12e0
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulUpdateResource.func1(0xc42f13e210, 0xc42f0351a0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1045 +0xd5
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc42f13e210, 0xc42f0351a0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc42078e360, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc42078e360, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x3d34530, 0xe, 0xc42078e360, 0xc4205d91f0, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420b63c20, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
<autogenerated>:1 +0x75
k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc42a0b7e00, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:93 +0x18a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4286f84c0, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4211dba40, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x3d372f1, 0xf, 0xc429455cb0, 0xc4211dba40, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc429e462c0, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
<autogenerated>:1 +0x75
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
net/http.HandlerFunc.ServeHTTP(0xc422da3b30, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc429e030c0, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc422da3c20, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc422da3d10, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc429e462e0, 0x7fa4a59aa370, 0xc42e1e0438, 0xc422e80500)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc429e46360, 0x986d800, 0xc42e1e0438, 0xc422e80500, 0xc42f03bda0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12`\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1aHrpc error: code = Unknown desc = proto: wrong wireType = 0 for field Key\"\x000\xf4\x03\x1a\x00\"\x00"

Host 2:

E0722 02:20:10.068483       1 watcher.go:268] failed to prepare current and previous objects: proto: wrong wireType = 0 for field WorkingDir
W0722 02:20:10.068573       1 reflector.go:341] storage/cacher.go:/pods: watch of *core.Pod ended with: Internal error occurred: proto: wrong
 wireType = 0 for field WorkingDir
...
E0722 02:20:10.758223       1 status.go:64] apiserver received an error that is not an metav1.Status: proto: wrong wireType = 0 for field Wor
kingDir
I0722 02:20:10.758561       1 wrap.go:42] GET /api/v1/pods: (3.060723ms) 500
goroutine 224226315 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc424092bd0, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:207 +0xdd
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc424092bd0, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:186 +0x35
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).WriteHeader(0xc433b005a0, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:188 +0xac
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.(*ResponseWriterDelegator).WriteHeader(0xc424f6ee70, 0x1f4)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:281 +0x45
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x3d863b2, 0x23, 0x7fd4dad1e2f8, 0xc42f8a7a8
0, 0x9868e80, 0xc4272851f0, 0xc425cdd900, 0x1f4, 0x984fc40, 0xc431159b00)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.
go:95 +0x8d
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x7fd4dacddbe8, 0xc424f6f3e0, 0x986b94
0, 0xc420cb6120, 0x0, 0x0, 0x3d209f1, 0x2, 0x9868e80, 0xc4272851f0, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.
go:123 +0x32b
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x7fd4dacddbe8, 0xc424f6f3e0, 0x9832b40, 0xc
42d900620, 0x986b940, 0xc420cb6120, 0x0, 0x0, 0x3d209f1, 0x2, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.
go:142 +0x163
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc430f3c3c0, 0x9832b40, 0xc42d900620, 0x9868e80, 0xc4272851f0, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:67 +0x10c
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x9868e80, 0xc4272851f0, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:257 +0xc5e
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulListResource.func1(0xc424f6ede0, 0xc425e89620)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1015 +0xd0
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc424f6ede0, 0xc425e89620)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:199 +0x208
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc42071f170, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:277 +0xb18
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(0xc42071f170, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199 +0x57
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x3d34530, 0xe, 0xc42071f170, 0xc4202eb500, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:152 +0x4e0
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc4202b6fa0, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
<autogenerated>:1 +0x75
k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc42830bb00, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:93 +0x18a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42275adc0, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x26d
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc42300ff80, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0xa1
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x3d372f1, 0xf, 0xc429ea4090, 0xc42300ff80, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:160 +0x6ad
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc429e50ee0, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
<autogenerated>:1 +0x75
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:52 +0x37d
n.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:78 +0x2b1
net/http.HandlerFunc.ServeHTTP(0xc4230ef400, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xcb
net/http.HandlerFunc.ServeHTTP(0xc429e50f00, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc429e50f80, 0x986d800, 0xc4272851c8, 0xc425cdd900, 0xc42738ed80)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:93 +0x8d
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:92 +0x1ab

logging error output: "k8s\x00\n\f\n\x02v1\x12\x06Status\x12F\n\x06\n\x00\x12\x00\x1a\x00\x12\aFailure\x1a.proto: wrong wireType = 0 for field WorkingDir\"\x000\xf4\x03\x1a\x00\"\x00"
[[Go-http-client/2.0] 10.87.136.119:60554]
HandlerFunc.ServeHTTP(0xc4230ef310, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:165 +0x42a
net/http.HandlerFunc.ServeHTTP(0xc429cfd880, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/usr/local/go/src/net/http/server.go:1918 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x203a
net/http.HandlerFunc.ServeHTTP(0xc4230ef360, 0x7fd4da66ade0, 0xc4272851c8, 0xc425cdd900)
...

What you expected to happen:
Did not expect the corrupted etcd record to be written.

How to reproduce it (as minimally and precisely as possible):
Unable to reproduce.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.10.3
• Cloud provider or hardware configuration: eks (aws)
• OS (e.g. from /etc/os-release): amazon-linux-2
• Others: etcd - 3.1.12