kubernetes: v1.8.6 cluster apiserver panic
Is this a BUG REPORT or FEATURE REQUEST?:
Uncomment only one, leave it on its own line:
/kind bug
/kind feature
What happened: I have a v1.8.6 cluster panic twice, I doubt where to pass an nil context.
// AuditEventFrom returns the audit event struct on the ctx
func AuditEventFrom(ctx Context) *audit.Event {
ev, _ := ctx.Value(auditKey).(*audit.Event)
return ev
}
apiserver log:
I0228 15:56:35.083908 1 trace.go:76] Trace[24533386]: "Update /apis/loadbalance.caicloud.io/v1alpha2/namespaces/kube-system/loadbalancers/de" (started: 2018-02-28 15:55:35.062227089 +0000 UTC) (total time: 1m0.021612612s):
Trace[24533386]: [1m0.021612612s] [1m0.021612612s] END
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x11a06bf]
goroutine 18221499 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.AuditEventFrom(0x0, 0x0, 0xc4261ce7c0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/context.go:158 +0x2f
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x0, 0x0, 0x9af2f00, 0xc4261ce7c0, 0xc42acc3d40, 0x17, 0xc424b09bb8, 0x8, 0x7fc37ef10658, 0xc431155950, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:108 +0xab
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.ErrorNegotiated(0x0, 0x0, 0x9ac6540, 0xc431fc8dc0, 0x9af2f00, 0xc4261ce7c0, 0xc42acc3d40, 0x17, 0xc424b09bb8, 0x8, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:132 +0x16b
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.(*RequestScope).err(0xc431ebc000, 0x9ac6540, 0xc431fc8dc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:80 +0x10e
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.UpdateResource.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/rest.go:865 +0x1578
k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.(*crdHandler).ServeHTTP(0xc420c22600, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/customresource_handler.go:219 +0xf1b
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4240e1c80, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc420909c70, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x48e7a9a, 0x17, 0xc4210d9200, 0xc420909c70, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420b6e820, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
<autogenerated>:69 +0x86
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc4252fde00, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:254 +0x26e
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4208fa8c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x48b77c4, 0xe, 0xc4210a2750, 0xc4208fa8c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc420acecc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
<autogenerated>:69 +0x86
k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc42bed0b00, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:91 +0x122
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc42bfc82c0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x3dd
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc4215e1650, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x72
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x48bc002, 0xf, 0xc426ae14d0, 0xc4215e1650, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:161 +0x301
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*director).ServeHTTP(0xc42422b500, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
<autogenerated>:69 +0x86
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:51 +0x37a
net/http.HandlerFunc.ServeHTTP(0xc421e39f90, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:98 +0x5a0
net/http.HandlerFunc.ServeHTTP(0xc4240e1dc0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:49 +0x21b2
net/http.HandlerFunc.ServeHTTP(0xc421c7a690, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:79 +0x2b0
net/http.HandlerFunc.ServeHTTP(0xc421c7a6e0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request.WithRequestContext.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/request/requestcontext.go:110 +0xef
net/http.HandlerFunc.ServeHTTP(0xc42422b520, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithCORS.func1(0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/cors.go:75 +0x189
net/http.HandlerFunc.ServeHTTP(0xc4239bcae0, 0x7fc37ef10658, 0xc431155950, 0xc42e13b600)
/usr/local/go/src/net/http/server.go:1942 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc42422b5a0, 0x9af6700, 0xc431155950, 0xc42e13b600, 0xc436ed9200)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:106 +0x8d
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x1ca
cc @sttts What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"6260bb08c46c31eea6cb538b34a9ceb3e406689c", GitTreeState:"clean", BuildDate:"2017-12-21T06:23:29Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
- Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 19 (16 by maintainers)
Commits related to this issue
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to kubernetes/kubernetes by deleted user 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to kubernetes/apiserver by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to kubernetes/kube-aggregator by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to kubernetes/sample-apiserver by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to kubernetes/apiextensions-apiserver by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to sttts/apiserver by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to sttts/kube-aggregator by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to sttts/sample-apiserver by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to sttts/apiextensions-apiserver by k8s-publishing-bot 6 years ago
- Merge pull request #60628 from sttts/sttts-global-requestcontextmapper-in-delegation Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the ins... — committed to openshift/kubernetes-sample-apiserver by k8s-publishing-bot 6 years ago
@hzxuzhonghu I believe this is due to multiple request context mappers, but only one WithRequestContext filter in the outer chain.