che: Devworkspace native auth on Kubernetes - Unauthorized! Configured service account doesn't have access

Describe the bug

We have a keycloak that authenticate agains GitLabs OIDC provider. (Informationen on our setup can be found here: https://github.com/eclipse/che/issues/20962#issuecomment-999655501) That actually served us well using Eclipse Che 7.40.

Upgrading to 7.42 seems to require native auth, so we tried to setup OAuth on Kubernetes using keycloak based on information provided by @sparkoo https://github.com/eclipse/che/issues/21036#issuecomment-1016095938. Are we missing something?

Dashboard just shows a blank page and shows the following errors.

image

It seems roles, bindings and serviceaccounts are not created. Should this be done by eclipse che in the provision api call?

Since chectl fails if oauth wasn’t properly setup: Is the “non-native-auth” way of handling users deprecated with the release of Che 7.42?

I get the idea of using OpenShifts buildin authentication and understand that some might find keycloak quite heavy compared to dex. But I think setting up OAuth on the Kubernetes API server isn’t really something a lot of users do on a regular basis. I’m not even sure if all managed kubernetes services allow changing these settings. Troubleshooting this tends to be quite difficult as well. (https://github.com/eclipse/che/issues/21036#issuecomment-1017712677)

Is setting nativeUserMode: false still a supported option?

Che version

7.42@latest

Steps to reproduce

I’ve created a new client kubernetes in keycloak and set the following settings on our kube-api-server:

oidc-client-id: kubernetes
oidc-issuer-url: 'https://auth.company.dev/auth/realms/git-dev'
oidc-username-claim: email

Chectl does not verify this setting as valid and throws a warning. Not sure what’s the issue, so tried bypassing the check --skip-oidc-provider-check. The chectl log file unfortunately does not contain aditional information.

Check if OIDC Provider installed...NOT INSTALLED
    → OIDC Provider is not installed in order to deploy Eclipse Che. To bypass OIDC Provider check use '--skip-oidc-provider-check' flag

Che configuration contains the following configuration.

  auth:
      externalIdentityProvider: true
      identityProviderURL: 'https://auth.company.dev/auth/realms/git-dev'
      openShiftoAuth: false
      oAuthClientName: 'dev-studio'
      oAuthSecret: 'db82*******'

Additional Info:

  • We’re running all services on a proper wildcard certificate.

Expected behavior

User native mode on kubernetes should work.

Runtime

Kubernetes (vanilla)

Screenshots

No response

Installation method

chectl/latest

Environment

Linux

Eclipse Che Logs

2022-01-21 12:20:49,422[nio-8080-exec-6]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2022-01-21 12:20:49
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.43.0.1/api/v1/namespaces/workspace-nm-33j87e/secrets/workspace-credentials-secret. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:623)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:493)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:475)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:807)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:188)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:155)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:88)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.configurator.CredentialsSecretConfigurator.configure(CredentialsSecretConfigurator.java:43)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.configureNamespace(KubernetesNamespaceFactory.java:570)
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.getOrCreate(KubernetesNamespaceFactory.java:334)
	at org.eclipse.che.workspace.infrastructure.kubernetes.provision.NamespaceProvisioner.provision(NamespaceProvisioner.java:42)
	at org.eclipse.che.workspace.infrastructure.kubernetes.api.server.KubernetesNamespaceService.provision(KubernetesNamespaceService.java:95)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:174)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:61)
	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:329)
	at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:319)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:257)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:131)
	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:61)
	at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:130)
	at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:62)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:777)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:161)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:119)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:353)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:872)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1705)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Unknown Source)

Additional context

Kubernetes: v1.20.4 Provisioned by Rancher

No errors in Kuberentes API Server logs.

OAuth2 Proxy Logs

[2022/01/21 12:17:06] [oauthproxy.go:862] No valid authentication in request. Initiating login.
10.42.3.0:51718 - 5906d68c8edb947f7082975011bdf5e5 - - [2022/01/21 12:17:06] che.company.dev GET - "/dashboard/assets/branding/manifest.json" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 302 363 0.000
10.42.3.0:52770 - 7c70b5c9fb6e52e5d6d9faebfb06e6c1 - n.m@company.com [2022/01/21 12:18:08] che.company.dev GET / "/api/keycloak/settings" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 404 88 0.004
10.42.3.0:52770 - b8a9a9c83a52fafd29537b4c0c40c021 - n.m@company.com [2022/01/21 12:18:11] che.company.dev GET / "/api/dex/settings" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 404 85 0.002
10.42.3.0:52770 - a029efa58feb4e2eadc4d8d8e9ebd9ef - n.m@company.com [2022/01/21 12:18:19] che.company.dev GET / "/api/oidc/settings" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 404 84 0.002
10.42.3.0:52770 - 52caf250ee8999ce9c0ffdf955b9be2c - n.m@company.com [2022/01/21 12:18:23] che.company.dev GET / "/api/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 200 726 0.015
10.42.3.0:52770 - db0ee253c8ccfe6226eb73950e1194c4 - n.m@company.com [2022/01/21 12:18:42] che.company.dev GET / "/api/oauth/settings" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 404 81 0.003
10.42.3.0:52770 - 5d02cc30fd50867f136bc7eb9bb346ca - n.m@company.com [2022/01/21 12:18:44] che.company.dev GET / "/api/oauth/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 200 28 0.003
10.42.3.0:52770 - 53d36dc1c20c58e961478856d84ebcb4 - n.m@company.com [2022/01/21 12:18:46] che.company.dev GET / "/api/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.62" 200 726 0.006
...

Since Che log tries to find a secret in the workspaces’ name:

$ kubectl get secrets --namespace workspace-nm-33j87e
NAME                  TYPE                                  DATA   AGE
default-token-ppmwz   kubernetes.io/service-account-token   3      110m

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 15 (5 by maintainers)

Most upvoted comments

For others reading this, setting everything up on rancher was quite simple.

Deploy keycloak

  • Deploy keycloak using an ssl certificate (e.g. https://auth.company.dev)
  • Create a realm (e.g. git-dev)
  • Create a client for kubernetes (e.g. kubernetes)

Note that this client must be used by che and kubernetes and must be protected. Otherweise OAuth2-Proxy will throw an error that secret must not be empty.

Setup OIDC for kube-api-server

On Rancher provisioned clusters the api server can be configured using extra_args.

  • In Rancher navigate to the edit cluster view.
  • Click on “Edit as as YAML” on the upper right. That’ll display all configuration settings
  • Set oidc-client-id, oidc-issuer-url, oidc-username-claim
kube-api:
  always_pull_images: false
  extra_args:
    oidc-client-id: kubernetes
    oidc-issuer-url: 'https://auth.company.dev/auth/realms/git-dev'
    oidc-username-claim: email

After saving the changes it takes a couple of minutes until all ctrl plane nodes are restarted. If using a self signed certificate, which we don’t, it gets a little tricky since kube api server must know the CAs certificate.

Configure Che

Set the external identifty provider and oidc username claim in the cr.

spec:
  server:
    customCheProperties:
      CHE_OIDC_USERNAME__CLAIM: "email"

  auth:
      externalIdentityProvider: true
      identityProviderURL: 'https://auth.company.dev/auth/realms/git-dev'
      openShiftoAuth: false
      oAuthClientName: 'kubernetes'
      oAuthSecret: '0...2'
+8
nils-mosbach on Jan 26, 2022

I ran into this same issue with Che 7.43 deployed via the che-operator 7.43 onto a GKE cluster with a separate Keycloak instance that I manage myself.

In case someone finds this helpful, here are the steps I took to finally get a working installation.

On GKE (probably EKS and AKS too) you cannot just change Kubernetes API server settings since this is being managed by the cloud provider.

  1. I had to enable GKE Identity Service so that I can authenticate users from Keycloak against Che. I chose to use the email claim to identify my users, so I have userClaim = email in its config. It is important to note here that this service seems to be implemented as an Envoy proxy in front of the Kubernetes API server, so now you have a new endpoint for Kubernetes API that will accept requests authenticated via an OIDC provider.
  2. As in the above comments, I also changed the Che username claim to email. The contents of the Che database were not important to me, so I dropped it and after restarting the che server, it started to populate the Name field of the usr table in the database with the email value from the Keycloak claim and also started to correctly create the Kubernetes RoleBindings in the user’s namespace with the user’s email as the Subject.
    customCheProperties:
      CHE_OIDC_USERNAME__CLAIM: "email"
  1. I had to get the che server to connect to the GKE Identity Service endpoint, instead of the standard Kubernetes one obtained from the in-cluster config by setting: CHE_INFRA_KUBERNETES_MASTER__URL=https://gke-oidc-envoy.anthos-identity-service
  2. The che-dashboard also makes some requests directly to Kubernetes which seem to be related to managing devworkspace objects. Here it was not possible to configure the master url, so as a quick and dirty hack, I forked it and changed the entrypoint script. I modified the standard Kubernetes environment variables so that the dashboard’s client would instead pick up the envoy proxy endpoint. This sucks, I know, but I was fed up at this point.
set -a

KUBERNETES_PORT=tcp://<gke-oidc-envoy LB IP>:443
KUBERNETES_PORT_443_TCP_ADDR=<gke-oidc-envoy LB IP>
KUBERNETES_PORT_443_TCP=tcp://<gke-oidc-envoy LB IP>:443
KUBERNETES_SERVICE_HOST=<gke-oidc-envoy LB IP>

set +a

Lots of effort was required before I was “CodeReady” 😛

Bonne chance!

+3
cristian-radu on Mar 15, 2022

@sparkoo could you please take a look?

+1
ibuziuk on Jan 21, 2022
Read more comments on GitHub
← che: Devfile registry deployment error 'standard_init_linux.go:211: exec user process caused "no such file or directory"'
che: [Dogfooding] Cannot build che-theia on che.openshift.io →