harbor: Login panic when using Redis HA with HAproxy

Expected behavior and actual behavior:

Sometimes, core service panic during login. Issue could not be reproduced when using single redis.

Could be related to #9841 but creating a separate issue because of https://github.com/goharbor/harbor/issues/9841#issuecomment-568843852

Steps to reproduce the problem: Please provide the steps to reproduce this problem.

Versions: Please specify the versions of following systems.

harbor version: v1.10.0
docker engine version: 18.9.2
kubelet version: 1.16.3

Additional context:

Log files:

Harbor:runtime error: invalid memory address or nil pointer dereference
Request Method:    GET
Request URL:    /c/oidc/callback?code=5I-o7wWFxcudwT8q18OR0D6qSzN5dZZEP07EzFU5yIn&state=dlap3hsued7f23qie9oxov833t5a556a
RemoteAddr:    172.25.14.0
Stack
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/harbor/src/vendor/github.com/astaxie/beego/controller.go:579
/harbor/src/core/api/base.go:180
/harbor/src/core/controllers/oidc.go:183
/usr/local/go/src/reflect/value.go:447
/usr/local/go/src/reflect/value.go:308
/harbor/src/vendor/github.com/astaxie/beego/router.go:815
/usr/local/go/src/net/http/server.go:2774
/usr/local/go/src/net/http/server.go:1878
/usr/local/go/src/runtime/asm_amd64.s:1337

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 8
Comments: 18 (2 by maintainers)

Most upvoted comments

We found out that beego uses a sessionExists function that just returns false in case of an error. We are running harbor with oidc auth and a ha redis backend.

After fixing this and adding multiple retries as well as idle connection checking to the redis client we haven’t seen this problem again. Since we only tested this since yesterday we can’t say for sure if the issue is really gone but at least #532 seems to be working for us again.

We try to add our changes into the next major update for beego. If you are building your own images you can test out these changes now already by adding this line to the go.mod file:

replace github.com/astaxie/beego => github.com/phiphi282/beego v1.12.2

I will follow up on this ticket in a few weeks to see if this really solved this issue.

phiphi282 on Jul 31, 2020

I can confirm in my case moving to a single standalone redis instance fixes the issue.

ninjaslothx on Mar 23, 2020