kubernetes: v1.8.6 cluster apiserver panic

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened: I have a v1.8.6 cluster panic twice, I doubt where to pass an nil context.

// AuditEventFrom returns the audit event struct on the ctx
func AuditEventFrom(ctx Context) *audit.Event {
	ev, _ := ctx.Value(auditKey).(*audit.Event)
	return ev
}

apiserver log:

I0228 15:56:35.083908       1 trace.go:76] Trace[24533386]: "Update /apis/loadbalance.caicloud.io/v1alpha2/namespaces/kube-system/loadbalancers/de" (started: 2018-02-28 15:55:35.062227089 +0000 UTC) (total time: 1m0.021612612s):
Trace[24533386]: [1m0.021612612s] [1m0.021612612s] END
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x11a06bf]

goroutine 18221499 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.AuditEventFrom(0x0, 0x0, 0xc4261ce7c0)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/context.go:158 +0x2f
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x0, 0x0, 0x9af2f00, 0xc4261ce7c0, 0xc42acc3d40, 0x17, 0xc424b09bb8, 0x8, 0x7fc37ef10658, 0xc431155950, ...)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:108 +0xab
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x0, 0x0, 0x9ac6540, 0xc431fc8dc0, 0x9af2f00, 0xc4261ce7c0, 0xc42acc3d40, 0x17, 0xc424b09bb8, 0x8, ...)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:132 +0x16b
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc431ebc000, 0x9ac6540, 0xc431fc8dc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:80 +0x10e
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.UpdateResource.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:865 +0x1578
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.(*crdHandler).ServeHTTP(0xc420c22600, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:219 +0xf1b
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4240e1c80, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc420909c70, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x48e7a9a, 0x17, 0xc4210d9200, 0xc420909c70, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420b6e820, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	<autogenerated>:69 +0x86
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4252fde00, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x26e
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4208fa8c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x48b77c4, 0xe, 0xc4210a2750, 0xc4208fa8c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420acecc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	<autogenerated>:69 +0x86
k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc42bed0b00, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x122
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42bfc82c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4215e1650, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x48bc002, 0xf, 0xc426ae14d0, 0xc4215e1650, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42422b500, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	<autogenerated>:69 +0x86
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37a
net/http.HandlerFunc.ServeHTTP(0xc421e39f90, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:98 +0x5a0
net/http.HandlerFunc.ServeHTTP(0xc4240e1dc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x21b2
net/http.HandlerFunc.ServeHTTP(0xc421c7a690, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b0
net/http.HandlerFunc.ServeHTTP(0xc421c7a6e0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef
net/http.HandlerFunc.ServeHTTP(0xc42422b520, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x189
net/http.HandlerFunc.ServeHTTP(0xc4239bcae0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
	/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc42422b5a0, 0x9af6700, 0xc431155950, 0xc42e13b600, 0xc436ed9200)
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x8d
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x1ca

cc @sttts What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 19 (16 by maintainers)

Commits related to this issue

Most upvoted comments

@hzxuzhonghu I believe this is due to multiple request context mappers, but only one WithRequestContext filter in the outer chain.