argo-workflows: Custom claim with oidc sso "HTTP ERROR 401"

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I’m trying to get SSO working with my org’s OIDC.

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# Branches: https://github.com/argoproj/argo-workflows/branches/
resources:
- github.com/argoproj/argo-workflows//manifests/cluster-install?ref=release-3.4.8
- ./argo-workflows-secrets-store.yaml
- ./argo-workflows-secrets.yaml
- ./argo-workflows-server-sa-readonly.yaml
- ./argo-workflows-clusterrole-readonly.yaml
- ./argo-workflows-clusterrolebinding-readonly.yaml

patches:
  - path: ./argo-workflows-cm.yaml
    target:
      kind: ConfigMap
      name: workflow-controller-configmap
  - path: ./argo-workflows-server-sa-admin.yaml
    target:
      kind: ServiceAccount
      name: argo-server
  - path: ./argo-workflows-server-deployment.yaml
    target:
      kind: Deployment
      name: argo-server

argo-workflows-server-deployment.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
spec:
  template:
    spec:
      containers:
        - name: argo-server
          args:
            - --loglevel
            - debug
            - --gloglevel
            - "6"
            - server
            - --auth-mode
            - sso

argo-workflows-server-sa-admin.yaml:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-server
  annotations:
    workflows.argoproj.io/rbac-rule: "'argo_worflows_admins' in groups"
    workflows.argoproj.io/rbac-rule-precedence: "1"

argo-workflows-cm.yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-workflows-cm
  namespace: argo
  labels:
    app.kubernetes.io/name: argo-workflows-cm
    app.kubernetes.io/part-of: argo
data:
  sso: |
    issuer: https://my.org.com/oidc/endpoint/default
    clientId:
      name: argo-workflows-secrets-oidc
      key: clientId
    clientSecret:
      name: argo-workflows-secrets-oidc
      key: clientSecret
    redirectUrl: https://argo-workflows-server.my.org.com/oauth2/callback
    scopes:
      - groups
      - email
      - profile
    rbac:
      enabled: true
    insecureSkipVerify: true
    customGroupClaimName: myCustomClaim

Starting up:

time="2023-06-22T17:01:22.355Z" level=debug msg="CLI version" version="{untagged 2023-05-30T13:31:45Z 58793a8ca54486c0a929ba7197d30b9f3cb3ce17 untagged clean go1.20.4 gc linux/amd64}"
time="2023-06-22T17:01:22.355Z" level=info msg="not enabling pprof debug endpoints"
I0622 17:01:22.355321       1 merged_client_builder.go:121] Using in-cluster configuration
I0622 17:01:22.355465       1 merged_client_builder.go:163] Using in-cluster namespace
time="2023-06-22T17:01:22.355Z" level=info authModes="[sso]" baseHRef=/ managedNamespace= namespace=argo secure=true ssoNamespace=argo
time="2023-06-22T17:01:22.355Z" level=info msg="Generating Self Signed TLS Certificates for Secure Mode"
I0622 17:01:23.374084       1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/argo/configmaps/workflow-controller-configmap 200 OK in 1017 milliseconds
I0622 17:01:23.383117       1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/argo/secrets/argo-workflows-secrets-oidc 200 OK in 5 milliseconds
I0622 17:01:23.805059       1 round_trippers.go:553] POST https://10.100.0.1:443/api/v1/namespaces/argo/secrets 409 Conflict in 44 milliseconds
I0622 17:01:23.809951       1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/argo/secrets/sso 200 OK in 4 milliseconds
time="2023-06-22T17:01:23.810Z" level=info msg="SSO configuration" clientId="{{argo-workflows-secrets-oidc} clientId <nil>}" insecureSkipVerify=true issuer="https://my.org.com/oidc/endpoint/default" issuerAlias=DISABLED redirectUrl="https://argo-workflows-server.my.org.com/oauth2/callback" scopes="[groups email profile openid]"
I0622 17:01:23.810589       1 reflector.go:219] Starting reflector *v1.ServiceAccount (20m0s) from pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167
I0622 17:01:23.810602       1 reflector.go:255] Listing and watching *v1.ServiceAccount from pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167
I0622 17:01:23.812702       1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/serviceaccounts?limit=500&resourceVersion=0 200 OK in 1 milliseconds
I0622 17:01:23.817408       1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/serviceaccounts?allowWatchBookmarks=true&resourceVersion=38164028&timeout=7m29s&timeoutSeconds=449&watch=true 200 OK in 1 milliseconds
I0622 17:01:23.910722       1 shared_informer.go:285] caches populated
time="2023-06-22T17:01:23.910Z" level=info msg="SSO enabled"
I0622 17:01:23.916583       1 round_trippers.go:553] GET https://10.100.0.1:443/api/v1/namespaces/argo/configmaps/workflow-controller-configmap 200 OK in 5 milliseconds
time="2023-06-22T17:01:23.916Z" level=info msg="Starting Argo Server" instanceID= version=untagged
time="2023-06-22T17:01:23.916Z" level=info msg="Creating event controller" asyncDispatch=false operationQueueSize=16 workerCount=4
time="2023-06-22T17:01:23.921Z" level=info msg="GRPC Server Max Message Size, MaxGRPCMessageSize, is set" GRPC_MESSAGE_SIZE=104857600
time="2023-06-22T17:01:23.921Z" level=info msg="Argo Server started successfully on https://localhost:2746" url="https://localhost:2746"
time="2023-06-22T17:01:41.559Z" level=info duration="102.201µs" method=GET path=index.html size=473 status=0
time="2023-06-22T17:01:45.592Z" level=info duration="65.141µs" method=GET path=/oauth2/redirect size=304 status=302
time="2023-06-22T17:02:01.110Z" level=info duration=986.410411ms method=GET path=/oauth2/callback size=0 status=401
time="2023-06-22T17:02:01.559Z" level=info duration="138.079µs" method=GET path=index.html size=473 status=0

I just can’t find where the problem lies. I’ve enabled debug logging, but as you can see, the logs don’t actually include debug logs for the authentication process.

I’ve tried troubleshooting with this Chrome extension: https://chrome.google.com/webstore/detail/saml-ws-federation-and-oa/hkodokikbjolckghdnljbkbhacbhpnkb. I can see that my request works, and my org’s OIDC provider responds successfully.

But I’m blind to seeing the claims come down to me from my org’s OIDC provider.

I’ve read through the docs over and over: https://argoproj.github.io/argo-workflows/argo-server-sso/

How can I troubleshoot this further??

Version

v3.4.8

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

Does not apply

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Does not apply

Logs from in your workflow’s wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

Does not apply