argo-workflows: Argo server logs print "token not valid for running mode" every 30s

Summary

The argo server logs are bombarded with the following error every 30s or so:

time="2021-04-17T23:23:10.983Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=WatchWorkflows grpc.service=workflow.WorkflowService grpc.start_time="2021-04-17T23:23:10Z" grpc.time_ms=0.064 span.kind=server system=grpc

Running with both SSO & client auth modes. Both work fine in UI & API calls (with authentication header with Bearer TOKEN) We haven’t seen this before v3 (upgraded from v2.12.11).

Diagnostics

Running on self-managed k8s cluster in AWS. Server behind an ingress, running insecure with TLS termination done in LB. Argo workflows v3.0.1

Server YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
  namespace: prod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: argo-server
  template:
    metadata:
      labels:
        app: argo-server
    spec:
      nodeSelector:
        general-use: 'true'
      containers:
        - name: argo-server
          image: argoproj/argocli:v3.0.1
          args:
            - server
            - --namespaced
            - --managed-namespace
            - workflows
            - --namespace
            - workflows
            - --configmap
            - workflow-controller-configmap
            - --auth-mode
            - sso
            - --auth-mode
            - client
            - --secure=false
            - --x-frame-options
            - SAMEORIGIN
          ports:
            - containerPort: 2746
              name: web
          securityContext:
            capabilities:
              drop:
                - ALL
          readinessProbe:
            httpGet:
              path: /
              port: 2746
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 20
          volumeMounts:
            - mountPath: /tmp
              name: tmp
      volumes:
        - name: tmp
          emptyDir: { }
      securityContext:
        runAsNonRoot: true
      serviceAccountName: argo-server-ui

SSO:

  sso: |
    issuer: https://accounts.google.com
    clientId:
      name: argo-secrets
      key: oidcClientId
    clientSecret:
      name: argo-secrets
      key: oidcClientSecret
    redirectUrl: https://our.redirect.domain/oauth2/callback
    scopes:
      - email
    rbac:
      enabled: true

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.