rancher: [BUG] Rancher crash with keycloak auth provider

Rancher Server Setup

  • Rancher version: v2.6.7-rc9
  • Installation option (Docker install/Helm Chart): v1.24.2_k3s2
  • Proxy/Cert Details: ingress/rancher

Information about the Cluster

  • Kubernetes version: v1.24.2
  • Cluster Type (Local/Downstream): Local

User Information

  • What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom)
    • If custom, define the set of permissions:

Describe the bug

The rancher pods crash with the following:

E0816 14:49:28.781073      33 runtime.go:79] Observed a panic: runtime.boundsError{x:1, y:1, signed:true, code:0x0} (runtime error: index out of range [1] with length 1)
goroutine 2931 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3f478c0, 0xc010d84150})
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0014d54f0})
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:49 +0x75
panic({0x3f478c0, 0xc010d84150})
        /usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.getSearchURL({0xc00a08fa40, 0x4})
        /go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:172 +0xe5
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).getFromKeyCloakByID(0xc00187a610, {0xc002e68ed4, 0x24}, {0xc002e68ecd, 0xeda8d942e}, 0xc00c58f0e0)
        /go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:135 +0xd3
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*keyCloakOIDCProvider).GetPrincipal(_, {_, _}, {{{0xc00187a599, 0x5}, {0xc001832a68, 0x17}}, {{0xc00187a600, 0xb}, {0xc00187a5c0, ...}, ...}, ...})
        /go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_provider.go:150 +0x2e7
github.com/rancher/rancher/pkg/auth/providers.GetPrincipal({_, _}, {{{0xc00187a599, 0x5}, {0xc001832a68, 0x17}}, {{0xc00187a600, 0xb}, {0xc00187a5c0, 0x6}, ...}, ...})
        /go/src/github.com/rancher/rancher/pkg/auth/providers/providers.go:164 +0x10e
github.com/rancher/rancher/pkg/auth/providerrefresh.(*refresher).refreshAttributes(0xc001476f00, 0xc0086c8580)
        /go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/refresher.go:256 +0xc08
github.com/rancher/rancher/pkg/auth/providerrefresh.RefreshAttributes(0xc0086c8580)
        /go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/daemon.go:83 +0x99
github.com/rancher/rancher/pkg/controllers/management/auth.(*UserAttributeController).sync(0xc00304c530, {0xc0014d5f40, 0x0}, 0xe)
        /go/src/github.com/rancher/rancher/pkg/controllers/management/auth/user_attribute_handler.go:39 +0x5f
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*userAttributeController).AddHandler.func1({0xc002dfad20, 0xc}, {0x418c100, 0xc0086c8580})
        /go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_user_attribute_controller.go:155 +0x42
github.com/rancher/norman/controller.(*genericController).AddHandler.func1({0xc002dfad20, 0xc}, {0x4b37fc8, 0xc0086c8580})
        /go/pkg/mod/github.com/rancher/norman@v0.0.0-20220627222520-b74009fac3ff/controller/generic_controller.go:60 +0x191
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc003060f40, {0xc002dfad20, 0x40d214}, {0x4b37fc8, 0xc0086c8580})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000fe04b0, {0xc002dfad20, 0xc}, {0x4b37fc8, 0xc0086c8580})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000a231e0, {0xc002dfad20, 0xc})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000a231e0, {0x37c5e40, 0xc0014d54f0})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000a231e0)
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fca57fcc328)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x4b09620, 0xc00380ec90}, 0x1, 0xc001cde960)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6
panic: runtime error: index out of range [1] with length 1 [recovered]
        panic: runtime error: index out of range [1] with length 1

goroutine 2931 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0014d54f0})
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:56 +0xd8
panic({0x3f478c0, 0xc010d84150})
        /usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.getSearchURL({0xc00a08fa40, 0x4})
        /go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:172 +0xe5
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*KeyCloakClient).getFromKeyCloakByID(0xc00187a610, {0xc002e68ed4, 0x24}, {0xc002e68ecd, 0xeda8d942e}, 0xc00c58f0e0)
        /go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_client.go:135 +0xd3
github.com/rancher/rancher/pkg/auth/providers/keycloakoidc.(*keyCloakOIDCProvider).GetPrincipal(_, {_, _}, {{{0xc00187a599, 0x5}, {0xc001832a68, 0x17}}, {{0xc00187a600, 0xb}, {0xc00187a5c0, ...}, ...}, ...})
        /go/src/github.com/rancher/rancher/pkg/auth/providers/keycloakoidc/keycloak_provider.go:150 +0x2e7
github.com/rancher/rancher/pkg/auth/providers.GetPrincipal({_, _}, {{{0xc00187a599, 0x5}, {0xc001832a68, 0x17}}, {{0xc00187a600, 0xb}, {0xc00187a5c0, 0x6}, ...}, ...})
        /go/src/github.com/rancher/rancher/pkg/auth/providers/providers.go:164 +0x10e
github.com/rancher/rancher/pkg/auth/providerrefresh.(*refresher).refreshAttributes(0xc001476f00, 0xc0086c8580)
        /go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/refresher.go:256 +0xc08
github.com/rancher/rancher/pkg/auth/providerrefresh.RefreshAttributes(0xc0086c8580)
        /go/src/github.com/rancher/rancher/pkg/auth/providerrefresh/daemon.go:83 +0x99
github.com/rancher/rancher/pkg/controllers/management/auth.(*UserAttributeController).sync(0xc00304c530, {0xc0014d5f40, 0x0}, 0xe)
        /go/src/github.com/rancher/rancher/pkg/controllers/management/auth/user_attribute_handler.go:39 +0x5f
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*userAttributeController).AddHandler.func1({0xc002dfad20, 0xc}, {0x418c100, 0xc0086c8580})
        /go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_user_attribute_controller.go:155 +0x42
github.com/rancher/norman/controller.(*genericController).AddHandler.func1({0xc002dfad20, 0xc}, {0x4b37fc8, 0xc0086c8580})
        /go/pkg/mod/github.com/rancher/norman@v0.0.0-20220627222520-b74009fac3ff/controller/generic_controller.go:60 +0x191
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc003060f40, {0xc002dfad20, 0x40d214}, {0x4b37fc8, 0xc0086c8580})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc000fe04b0, {0xc002dfad20, 0xc}, {0x4b37fc8, 0xc0086c8580})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000a231e0, {0xc002dfad20, 0xc})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000a231e0, {0x37c5e40, 0xc0014d54f0})
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000a231e0)
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(...)
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fca57fcc328)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x4b09620, 0xc00380ec90}, 0x1, 0xc001cde960)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
        /go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6

To Reproduce

I’m not sure what the cause is because the initial creation of the keycloak auth provider and login is successful. There would be one pod still running and the other two would be on a crash loop.

Result

Expected Result

Screenshots

Additional context

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 10
  • Comments: 15 (1 by maintainers)

Most upvoted comments

I came up with a solution for my self managed KeyCloak instance (Bitnami Helm Chart):

Here are the relevant helm values:

image:
  tag: 20.0.2-debian-11-r9
proxy: edge
httpRelativePath: /auth/
ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "KC-NGINX-SESSION"
    nginx.ingress.kubernetes.io/app-root: /auth
  tls: true
  • httpRelativePath prefixes the /auth/ path again
  • nginx.ingress.kubernetes.io/app-root: /auth tells nginx to create a 301 (moved permanently) from / to /auth
+2
toms-place on Jan 9, 2023

https://github.com/rancher/rancher/blob/v2.6.7/pkg/auth/providers/keycloakoidc/keycloak_client.go#L167-L174

seems to be the problem with fresh keycloak install > keycloak17. The standard url does not contain the auth prefix anymore.

Related to #38480, duplicated by #38683

+1
bpedersen2 on Aug 23, 2022
Read more comments on GitHub
← rancher: [BUG] Rancher 2.7.5 Active Directory authentication fails completely in Rancher 2.7.5 due to wrong objectGUID escaping
rancher: [BUG] rancher keeps logging errors →