harbor: [v1.10.0] Panic 500 when pushing image (using oidc authentication)

Hi Harbor team,

I have a production environment which has 5 harbor nodes, I always find some panic error in 2 of them, all the configuration files are same, my staff also report some 500 error when thay are pushing image, here’s some panic logs which I found in core.log:

Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] the request url is  /service/token#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] Handler crashed with error interface conversion: interface {} is nil, not models.OIDCSetting#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /usr/local/go/src/runtime/panic.go:522#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /usr/local/go/src/runtime/iface.go:248#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /harbor/src/common/utils/oidc/helper.go:243#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /harbor/src/common/utils/oidc/secret.go:116#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /harbor/src/common/utils/oidc/secret.go:131#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /harbor/src/core/filter/security.go:242#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /harbor/src/core/filter/security.go:135#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/router.go:615#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/router.go:701#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /usr/local/go/src/net/http/server.go:2774#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /usr/local/go/src/net/http/server.go:1878#033[0m
Mar  2 17:31:36 172.17.0.1 core[1051]: 2020/03/02 09:31:36 #033[1;35m[C] [panic.go:522] /usr/local/go/src/runtime/asm_amd64.s:1337#033[0m

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 3
Comments: 15 (2 by maintainers)

Commits related to this issue

fix #10913: initialize provider before calling Load — committed to thoro/harbor by thoro 4 years ago
fix #10913: initialize provider before calling Load Signed-off-by: Thoro <thomas@ccs.at> — committed to thoro/harbor by thoro 4 years ago
fix #10913: initialize provider before calling Load Signed-off-by: Thomas Rosenstein <thomas.rosenstein@creamfinance.com> — committed to thoro/harbor by thoro 4 years ago
fix #10913: initialize provider before calling Load Signed-off-by: Thomas Rosenstein <thomas.rosenstein@creamfinance.com> — committed to thoro/harbor by thoro 4 years ago
Merge pull request #13126 from Thoro/fix-10913-oidc-error-after-restart fix #10913: initialize oidc provider before calling Load — committed to goharbor/harbor by ywk253100 4 years ago

Most upvoted comments

@wglambert @ragarcia26 @burdzwastaken
I can reproduce this panic problem. I have 6 nodes in my harbor cluster at this moment, everytime I run ‘docker-compose down && docker-compose up -d’ to recreate harbor, I need to use every node to login harbor, otherwise the panic error occurs (modify hosts file to point my harbor domain to specified node).

I looked into source code, found this file: https://github.com/goharbor/harbor/blob/master/src/common/utils/oidc/helper.go
line 42:

type providerHelper struct {
	sync.Mutex
	instance     atomic.Value
	setting      atomic.Value
	creationTime time.Time
}

wonder if it is the atomic value cause the problem? every node need to connect OIDC provider in order to init models.OIDCSetting？

joeyjy on Apr 20, 2020

this change has been merged to master since October. Any idea when it’ll be packaged in a release ??

muzimuzi42 on Dec 30, 2020

@steven-zou can we get some labels on this issue for visibility as this is reproducible in all versions since at least 1.10 and affects service availability (autoscaling, HA, etc) for instances using OIDC auth. 🙏

horjulf on Sep 22, 2020