rancher: [BUG] Potential panic in keycloakoidc client caused by uncaught error

Rancher Server Setup

  • Rancher version: v2.6.7
  • Installation option (Docker install/Helm Chart): Docker
  • Proxy/Cert Details: no proxy, self-sigend cert

Describe the bug rancher with keycloak auth configured constantly crashes with the following backtrace:

2022/08/22 06:35:05 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
E0822 06:35:05.202301      52 runtime.go:79] Observed a panic: runtime.boundsError{x:1, y:1, signed:true, code:0x0} (runtime error: index out of range [1] with length 1)
goroutine 5333 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3f478c0, 0xc005166120})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0009088c0})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:49 +0x75
panic({0x3f478c0, 0xc005166120})
	/usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.getSearchURL({0xc00537af90, 0x4})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:172 +0xe5
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).getFromKeyCloakByID(0xc00260a998, {0xc0019fc6d4, 0x24}, {0xc0019fc6cd, 0xeda95187f}, 0xc006571a40)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:135 +0xd3
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*keyCloakOIDCProvider).GetPrincipal(_, {_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, ...}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_provider.go:150 +0x2e7
github.com/rancher/rancher/pkg/auth/providers.GetPrincipal({_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, 0x6}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/providers.go:164 +0x10e
github.com/rancher/rancher/pkg/auth/providerrefresh.(*refresher).refreshAttributes(0xc002c7f780, 0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/refresher.go:256 +0xc08
github.com/rancher/rancher/pkg/auth/providerrefresh.RefreshAttributes(0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/daemon.go:83 +0x99
github.com/rancher/rancher/pkg/controllers/management/auth.(*UserAttributeController).sync(0xc006fc1c30, {0xc00713feb0, 0x0}, 0x696c6163203a656d)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/auth/user_attribute_handler.go:39 +0x5f
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*userAttributeController).AddHandler.func1({0xc00274ee10, 0xa}, {0x418c100, 0xc001274580})
	/go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_user_attribute_controller.go:155 +0x42
github.com/rancher/norman/controller.(*genericController).AddHandler.func1({0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20220627222520-b74009fac3ff/controller/generic_controller.go:60 +0x191
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc001064860, {0xc00274ee10, 0x40d214}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000c17e00, {0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000aafc30, {0xc00274ee10, 0xa})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000aafc30, {0x37c5e40, 0xc0009088c0})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000aafc30)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f630febe2a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00399f680, {0x4b09600, 0xc0044b6480}, 0x1, 0xc0019d01e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0039a0960, 0x3b9aca00, 0x0, 0xf0, 0xc0039a10e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0xc0039a1680, 0xc0039a1860, 0xc0008fee00)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6
panic: runtime error: index out of range [1] with length 1 [recovered]
	panic: runtime error: index out of range [1] with length 1

goroutine 5333 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0009088c0})
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:56 +0xd8
panic({0x3f478c0, 0xc005166120})
	/usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.getSearchURL({0xc00537af90, 0x4})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:172 +0xe5
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).getFromKeyCloakByID(0xc00260a998, {0xc0019fc6d4, 0x24}, {0xc0019fc6cd, 0xeda95187f}, 0xc006571a40)
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:135 +0xd3
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*keyCloakOIDCProvider).GetPrincipal(_, {_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, ...}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_provider.go:150 +0x2e7
github.com/rancher/rancher/pkg/auth/providers.GetPrincipal({_, _}, {{{0xc00260a95a, 0x5}, {0xc0032e3680, 0x17}}, {{0xc00260a9a0, 0xb}, {0xc00260a960, 0x6}, ...}, ...})
	/go/src/github.com/rancher/rancher/pkg/auth/providers/providers.go:164 +0x10e
github.com/rancher/rancher/pkg/auth/providerrefresh.(*refresher).refreshAttributes(0xc002c7f780, 0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/refresher.go:256 +0xc08
github.com/rancher/rancher/pkg/auth/providerrefresh.RefreshAttributes(0xc001274580)
	/go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/daemon.go:83 +0x99
github.com/rancher/rancher/pkg/controllers/management/auth.(*UserAttributeController).sync(0xc006fc1c30, {0xc00713feb0, 0x0}, 0x696c6163203a656d)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/auth/user_attribute_handler.go:39 +0x5f
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*userAttributeController).AddHandler.func1({0xc00274ee10, 0xa}, {0x418c100, 0xc001274580})
	/go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_user_attribute_controller.go:155 +0x42
github.com/rancher/norman/controller.(*genericController).AddHandler.func1({0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20220627222520-b74009fac3ff/controller/generic_controller.go:60 +0x191
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc001064860, {0xc00274ee10, 0x40d214}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000c17e00, {0xc00274ee10, 0xa}, {0x4b37fa8, 0xc001274580})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000aafc30, {0xc00274ee10, 0xa})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000aafc30, {0x37c5e40, 0xc0009088c0})
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000aafc30)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f630febe2a0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00399f680, {0x4b09600, 0xc0044b6480}, 0x1, 0xc0019d01e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0039a0960, 0x3b9aca00, 0x0, 0xf0, 0xc0039a10e0)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0xc0039a1680, 0xc0039a1860, 0xc0008fee00)
	/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 7
  • Comments: 22 (9 by maintainers)

Commits related to this issue

Most upvoted comments

For people whose rancher2 is crash-looping because of this bug, here is how to remove the config and allows rancher to start: kubectl patch -n cattle-system AuthConfig keycloakoidc --type='merge' -p '{"enabled":false, "authEndpoint":null, "issuer":null}'

+2
Elektordi on Mar 3, 2023

Verified upgrade from v2.7.0 to v2.7-head 9376ad2

  • Created a keycloak server on v19.0.2
  • Created a rancher server on v2.7.0
  • Enabled keycloak OIDC on the rancher server
  • Created a downstream cluster and added user from keycloak as cluster owner
  • Upgrade rancher server and verified was able to login without any error via keycloak
  • No panic seen in the rancher logs.
  • Created a new cluster rke2 downstream node driver 1 etcd+cp, 1 worker node and verified no panics seen in the rancher logs.
+1
anupama2501 on Feb 16, 2023

Release Note:

Credit goes to github user @jamhed for developing a fix to this issue.

Rancher attempted to form a search URL for keycloak by splitting based on the pattern of /auth/. Newer keycloak versions, namely those using the Quarkus Distribution, did not include this value, causing a panic to occur when users attempted to integrate keycloak with Rancher. Rancher has been updated to properly form this search url for Quarkus Based distributions.

+1
MbolotSuse on Mar 10, 2023

It appears there might be workaround for those that are able to modify the Keycloak server base configuration:

Changing http-relative-path from / to /auth/ should result in Keycloak using backwards compatible URLs for the autodiscovery endpoints. https://www.keycloak.org/server/all-config

*Haven’t confirmed that to be working

+1
janeczku on Nov 2, 2022
Read more comments on GitHub
← rancher: [BUG] Multiple server nodes pre-drains in an RKE2 upgrade
rancher: Bug: Project shows "Cluster not ready" →