argo-workflows: SSO authentication not working

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I upgraded Argo Workflow from 3.1.13 to 3.4.3. SSO Authentication was working fine with 3.1.13; however, the 3.4.3 doesn’t seem to work. The SSO configuration (Okta) has not changed.

When I tried to open the UI and click on login on the SSO, I get a red banner on the down right corner saying Failed to load version/info Error: Unauthorized. After that the web page just tries to load and after sometime it replies with test-ce-argo-server-integration.k8s.cnqr.tech didn't send any data.ERR_EMPTY_RESPONSE

Version

3.4.3

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

N/A - The issue happens at login time, so I can't run any workflow.

Logs from the workflow controller

I am attaching the logs of the workflow server, because the error happens during authentication:

time="2022-11-07T13:37:32.115Z" level=info duration=2.752873ms method=GET path=/main.2430295409b8b54e52ad.js size=1471060 status=0
time="2022-11-07T13:37:32.117Z" level=info duration="23.156µs" method=GET path=index.html size=0 status=304
time="2022-11-07T13:37:33.935Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetUserInfo grpc.service=info.InfoService grpc.start_time="2022-11-07T13:37:33Z" grpc.time_ms=0.047 span.kind=server system=grpc
time="2022-11-07T13:37:33.935Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2022-11-07T13:37:33Z" grpc.time_ms=0.028 span.kind=server system=grpc
time="2022-11-07T13:37:33.935Z" level=info duration=1.628208ms method=GET path=/api/v1/userinfo size=56 status=401
time="2022-11-07T13:37:33.935Z" level=info duration=2.284157ms method=GET path=/api/v1/info size=56 status=401
time="2022-11-07T13:37:34.116Z" level=info duration="202.575µs" method=GET path=/assets/fonts/fa-solid-900.woff2 size=150472 status=0
time="2022-11-07T13:37:34.116Z" level=info duration="111.014µs" method=GET path=/assets/images/logo.png size=41464 status=0
time="2022-11-07T13:37:34.389Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=CollectEvent grpc.service=info.InfoService grpc.start_time="2022-11-07T13:37:34Z" grpc.time_ms=0.03 span.kind=server system=grpc
time="2022-11-07T13:37:34.389Z" level=info duration="438.543µs" method=POST path=/api/v1/tracking/event size=56 status=401
time="2022-11-07T13:37:36.819Z" level=info duration="67.668µs" method=GET path=index.html size=473 status=0
time="2022-11-07T13:37:56.819Z" level=info duration="68.792µs" method=GET path=index.html size=473 status=0
time="2022-11-07T13:38:16.819Z" level=info duration="74.393µs" method=GET path=index.html size=473 status=0
time="2022-11-07T13:38:36.819Z" level=info duration="83.064µs" method=GET path=index.html size=473 status=0
time="2022-11-07T13:38:56.819Z" level=info duration="68.599µs" method=GET path=index.html size=473 status=0
time="2022-11-07T13:39:16.819Z" level=info duration="81.42µs" method=GET path=index.html size=473 status=0
time="2022-11-07T13:39:36.819Z" level=info duration="72.698µs" method=GET path=index.html size=473 status=0

Logs from in your workflow’s wait container

N/A

This is the service account configured for RBAC:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  annotations:
    # The rule is an expression used to determine if this service account
    # should be used.
    # * `groups` - an array of the OIDC groups
    # * `iss` - the issuer ("argo-server")
    # * `sub` - the subject (typically the username)
    # Must evaluate to a boolean.
    # If you want an account to be the default to use, this rule can be "true".
    # Details of the expression language are available in
    # https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md.
    workflows.argoproj.io/rbac-rule: "true"
    # The precedence is used to determine which service account to use whe
    # Precedence is an integer. It may be negative. If omitted, it defaults to "0".
    # Numerically higher values have higher precedence (not lower, which maybe
    # counter-intuitive to you).
    # If two rules match and have the same precedence, then which one used will
    # be arbitrary.
    workflows.argoproj.io/rbac-rule-precedence: "0"