kubernetes: Kubernetes API-Server doesn't start using OIDC

Hello everyone,

I think I am facing an issue, similar to #64206. When adding OIDC (Keycloak), running inside the affected On-Prem Kubernetes cluster, I receive errors, when switching from client type public to confidential(requiring a client secert). Once re-configured, API Server refuses connections. Further, port of Master Node is not in listening mode. (unused):

kubectl get nodes
The connection to the server 192.168.190.115:6443 was refused - did you specify the right host or port?

fabiansc@Kubernetes-Master:~$ sudo netstat -tulpn | grep LISTEN | grep :6443
fabiansc@Kubernetes-Master:~$

less +F /var/log/syslog
Jan  6 01:06:11 Kubernetes-Master kubelet[1707]: W0106 01:06:11.751670    1707 status_manager.go:550] Failed to get status for pod "kube-scheduler-kubernetes-master_kube-system(ee4c94eb845abf1878fb3c4c489b1365)": Get "https://192.168.190.115:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-kubernetes-master": dial tcp 192.168.190.115:6443: connect: connection refused
Jan  6 01:06:11 Kubernetes-Master kubelet[1707]: W0106 01:06:11.752072    1707 status_manager.go:550] Failed to get status for pod "calico-node-8cjds_calico-system(16e05678-d91f-4c2e-810e-5913d1c88ca9)": Get "https://192.168.190.115:6443/api/v1/namespaces/calico-system/pods/calico-node-8cjds": dial tcp 192.168.190.115:6443: connect: connection refused
Jan  6 01:06:11 Kubernetes-Master kubelet[1707]: W0106 01:06:11.752287    1707 status_manager.go:550] Failed to get status for pod "calico-kube-controllers-7487d7f956-z4gl9_calico-system(ab04baa9-98c3-42c9-ac68-f001f29b9065)": Get "https://192.168.190.115:6443/api/v1/namespaces/calico-system/pods/calico-kube-controllers-7487d7f956-z4gl9": dial tcp 192.168.190.115:6443: connect: connection refused
[...]

I faced this issue while enabling RBAC for Kiali (see Kiali Discussion #3557 for many details).

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Please note, that Keycloak is running on an official DigiCert SSL Certificacte (1 year runtime). Even thoug my nodes (neither master, nor slaves) trusted DigiCert for any reasons (I added it to the Ubuntu root-ca trust). In case API-Server needs to “load” the DigiCert certificate as well, a short hint for mounting it is appreciated.

As #64206 requested a certain curl:

curl --cacert /usr/share/ca-certificates/extra/keycloak.crt https://keycloak.example.de:30000/auth/realms/***
{"realm":"***","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmISx2ALYm3rCMNlH8N5XfIgNts1SKFrYPsIwuKwPxql1R26U/BV4EdgvoqHXLtoLAwHIJ2cXqQ8V4CfVeFpCMMMeUUCNbH4qWtGL+R5FfudywjhS8SP7H1tpt+kz2MuWoMe+PZ3mx+rzaCy+gaN+MH08F1pDN8vzzdWTXVs7TFt3eeqQl4Snc+gSwzj6/YstwNFmQ8TcbqqxVB1rcx5gkDPf0W9Y7YAJor9HCoJWRE9ahgpNgOq/k5cmHoLqCkaDXtZtiSZSZfqUmdBWJ/X3On07uyntgVmCjk/mPwXAe9gL+zRawBKo9tS0+OH+iulGN/+LOX8fbj7dQC1zo1aedQIDAQAB","token-service":"https://keycloak.example.de:30000/auth/realms/***/protocol/openid-connect","account-service":"https://keycloak.example.de:30000/auth/realms/***/account","tokens-not-before":0}


curl https://keycloak.example.de:30000/auth/realms/**** -v
*   Trying 84.184.171.193...
* TCP_NODELAY set
* Connected to keycloak.example.de (****) port 30000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.example.de
*  start date: Sep 26 00:00:00 2020 GMT
*  expire date: Oct  3 12:00:00 2021 GMT
*  subjectAltName: host "keycloak.example.de" matched cert's "*.example.de"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=Encryption Everywhere DV TLS CA - G1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x557967b9b5c0)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /auth/realms/**** HTTP/2
> Host: keycloak.example:30000
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 200
< cache-control: no-cache
< x-xss-protection: 1; mode=block
< x-frame-options: SAMEORIGIN
< referrer-policy: no-referrer
< date: Tue, 05 Jan 2021 20:58:15 GMT
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< content-type: application/json
< content-length: 662
< x-envoy-upstream-service-time: 3
< server: istio-envoy
<
* Connection #0 to host keycloak.example.de left intact
{"realm":"****","public_key":"MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAmISx2ALYm3rCMNlH8N5XfIgNts1SKFrYPsIwuKwPxql1R26U/BV4EdgvoqHXLtoLAwHIJ2cXqQ8V4CfVeFpCMMMeUUCNbH4qWtGL+R5FfudywjhS8SP7H1tpt+kz2MuWoMe+PZ3mx+rzaCy+gaN+MH08F1pDN8vzzdWTXVs7TFt3eeqQl4Snc+gSwzj6/YstwNFmQ8TcbqqxVB1rcx5gkDPf0W9Y7YAJor9HCoJWRE9ahgpNgOq/k5cmHoLqCkaDXtZtiSZSZfqUmdBWJ/X3On07uyntgVmCjk/mPwXAe9gL+zRawBKo9tS0+OH+iulGN/+LOX8fbj7dQC1zo1aedQIDAQAB","token-service":"https://keycloak.example.de:30000/auth/realms/***/protocol/openid-connect","account-service":"https://keycloak.example.de:30000/auth/realms/***/account","tokens-not-before":0}

sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
can you please also check if group and user prefix is configured correctly?

spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-log-path=/var/log/kube-apiserver.log # does not work for any reason
    - --advertise-address=192.168.190.115
    - --allow-privileged=true
#    - --authorization-mode=Node,RBAC
    - --oidc-issuer-url=https://keycloak.example:30000/auth/realms/***
    - --oidc-client-id=kubernetes-cluster
    - --oidc-client-secret=***
    - --oidc-username-claim=email
    - --oidc-groups-claim=groups
    - --oidc-groups-prefix=oidc:
    - --oidc-username-prefix=oidc:
    - --oidc-ca-file=/usr/share/ca-certificates/extra/keycloak.crt
[...]

cat /var/log/kube-apiserver.log
cat: /var/log/kube-apiserver.log: Datei oder Verzeichnis nicht gefunden

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (7 by maintainers)

Most upvoted comments

I think I got it:

    - "--oidc-groups-prefix=oidc:"
    - "--oidc-username-prefix=oidc:"

That is correct

liggitt on Jan 6, 2021

I scripted my login-process using CLI-Tool from Toshiaki Maki. Looks like I had an mismatching user within my kubectl conf or time-condition for ID-Token.

Now I get a new error, due to incorrect RoleBinding:

kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "\"oidc:\"test@gmail.com" cannot list resource "nodes" in API group "" at the cluster scope

I will deploy a mapping for my Keycloak Group Platform Administrators:

vim crb.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: oidc-cluster-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: "oidc:/Platform Administrators"
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: oidc:test@gmail.com

kubectl apply -f crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/oidc-cluster-admins configured

This is ClusterRoleBinding references:

kubectl describe clusterrole cluster-admin
Name:         cluster-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
  Resources  Non-Resource URLs  Resource Names  Verbs
  ---------  -----------------  --------------  -----
  *.*        []                 []              [*]
             [*]                []              [*]

However; I still get:

kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "\"oidc:\"test@gmail.com" cannot list resource "nodes" in API group "" at the cluster scope

It looks like it correlates with my API-Server prefix. How can I add oidc: as prefix (without " ")?

spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-log-path=/var/log/kube-apiserver.log
    - --advertise-address=192.168.190.115
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --oidc-issuer-url=https://keycloak.example.de:30000/auth/realms/***
    - --oidc-client-id=kubernetes-cluster
    - --oidc-username-claim=email
    - --oidc-groups-claim=groups
    - --oidc-groups-prefix="oidc:"
    - --oidc-username-prefix="oidc:"

I think I got it:

    - "--oidc-groups-prefix=oidc:"
    - "--oidc-username-prefix=oidc:"

Can someone confirm this notation is intended for /etc/kubernetes/manifests/kube-apiserver.yaml?

schroedt on Jan 6, 2021