argo-cd: Failed to query provider "https://argocd-host/api/dex": 502 Bad Gateway:

I reached-out about this issue in the ArgoCD slack but received no reply.

Checklist:

• I’ve searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• I’ve included steps to reproduce the bug.
• I’ve pasted the output of argocd version.

Describe the bug

I’ve gotten ArgoCD v1.6.1 deployed to an EKS cluster running v1.15.11-eks-af3caf along with Istio 1.4.7 successfully. Now i’m trying to get SAML auth enabled with Dex via Okta as described in the documentation. However, when I click on the Login via Okta button at the login page I’m immediately met with an error page stating: Failed to query provider "https://argocd-host/api/dex": 502 Bad Gateway: Attempting to authenticate via the CLI returns the same error as well.

I’ve dug through the documentation and the issues here on github and haven’t been able to find much but have ensured that the data.url config parameter is set correctly. I also haven’t been able to find anyone through general googling that seems to have the same sort of setup with running EKS, istio ingress and AWS NLB, so I’m not sure if perhaps there’s something about this particular combination that’s screwing things up. I do have a few other services/apps running in this same cluster (grafana, jenkins, etc.) that are using Okta authentication without issue, so I don’t believe there’s any sort of ingress/egress rules getting in the way of things.

To Reproduce

Have Istio 1.4.7 running in an EKS 1.15 cluster deployed with a configuration very similar to what’s documented in this issue with the --insecure flag set on argocd-server. The insecure flag is set because I am terminating at the AWS NLB. Configure the argocd-cm ConfigMap as the following:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  url: https://argocd-host
  redirectURL: https://argocd-host/api/dex/callback
  dex.config: |
    logger:
      level: debug
      format: json
    connectors:
    - type: saml
      id: okta
      name: Okta
      config:
        ssoURL: https://<myorgsoktassourl>/sso/saml
        redirectURI: https://argocd-host/api/dex/callback
        caData: <bas64-encoded cert>
        usernameAttr: email
        emailAttr: email
        groupsAttr: groups

Go to the configured ArgoCD URL of https://argocd-host and click on the Login Via Okta button that now appears.

Expected behavior

Authentication should happen successfully via Okta and I should be taken to the Applications page on my ArgoCD instance.

Screenshots

I don’t think a screenshot is necessary but the error message displayed after trying login via okta is:

Failed to query provider "https://argocd-host/api/dex": 502 Bad Gateway: 

Version

argocd: v1.6.0+c10ae24
  BuildDate: 2020-06-16T22:41:56Z
  GitCommit: c10ae246ab02f1356147118a1979fedcd1ceb704
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.6.1+159674e
  BuildDate: 2020-06-19T00:41:05Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

Logs I have enabled debug logging for the running services, though I didn’t too many more log entries as a result

argocd-server log:

{"level":"info","msg":"Configmap/secret informer synced","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"Starting configmap/secret informers","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"configmap informer cancelled","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"secrets informer cancelled","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"Configmap/secret informer synced","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"Starting configmap/secret informers","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"secrets informer cancelled","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"configmap informer cancelled","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"Configmap/secret informer synced","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"Starting configmap/secret informers","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"configmap informer cancelled","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"secrets informer cancelled","time":"2020-07-21T21:33:35Z"}
{"level":"info","msg":"Configmap/secret informer synced","time":"2020-07-21T21:33:36Z"}
{"level":"info","msg":"Creating client app (argo-cd)","time":"2020-07-21T21:33:36Z"}
{"level":"info","msg":"argocd v1.6.1+159674e serving on port 8080 (url: https://argocd-host, tls: false, namespace: argocd, sso: true)","time":"2020-07-21T21:33:36Z"}
{"level":"info","msg":"0xc000243260 subscribed to settings updates","time":"2020-07-21T21:33:36Z"}
{"level":"info","msg":"Starting rbac config informer","time":"2020-07-21T21:33:36Z"}
{"level":"info","msg":"RBAC ConfigMap 'argocd-rbac-cm' added","time":"2020-07-21T21:33:36Z"}
{"grpc.method":"Version","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"version.VersionService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /version.VersionService/Version","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"dir":"","execID":"SS8qK","level":"info","msg":"ks  version","time":"2020-07-21T21:34:17Z"}
{"duration":67656209,"execID":"SS8qK","level":"debug","msg":"ksonnet version: 0.13.1\njsonnet version: v0.11.2\nclient-go version: kubernetes-1.10.4\n","time":"2020-07-21T21:34:17Z"}
{"dir":"","execID":"zmljB","level":"info","msg":"kustomize version","time":"2020-07-21T21:34:17Z"}
{"duration":48298630,"execID":"zmljB","level":"debug","msg":"{Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}\n","time":"2020-07-21T21:34:17Z"}
{"dir":"","execID":"H6uda","level":"info","msg":"helm version --client","time":"2020-07-21T21:34:17Z"}
{"duration":50177755,"execID":"H6uda","level":"debug","msg":"version.BuildInfo{Version:\"v3.2.0\", GitCommit:\"e11b7ce3b12db2941e90399e874513fbd24bcb71\", GitTreeState:\"clean\", GoVersion:\"go1.13.10\"}\n","time":"2020-07-21T21:34:17Z"}
{"dir":"","execID":"VSlpQ","level":"info","msg":"kubectl version --client","time":"2020-07-21T21:34:17Z"}
{"duration":62498984,"execID":"VSlpQ","level":"debug","msg":"Client Version: version.Info{Major:\"1\", Minor:\"14\", GitVersion:\"v1.14.0\", GitCommit:\"641856db18352033a0d96dbc99153fa3b27298e5\", GitTreeState:\"clean\", BuildDate:\"2019-03-25T15:53:57Z\", GoVersion:\"go1.12.1\", Compiler:\"gc\", Platform:\"linux/amd64\"}\n","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"Version","grpc.service":"version.VersionService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":229.519,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.method":"Get","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"cluster.SettingsService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /cluster.SettingsService/Get","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"Get","grpc.service":"cluster.SettingsService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":1.049,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"error":"rpc error: code = Unauthenticated desc = no session information","grpc.code":"Unauthenticated","grpc.method":"List","grpc.service":"cluster.ClusterService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":0.341,"level":"info","msg":"finished unary call with code Unauthenticated","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.method":"GetUserInfo","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /session.SessionService/GetUserInfo","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"GetUserInfo","grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":0.56,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"error":"rpc error: code = Unauthenticated desc = no session information","grpc.code":"Unauthenticated","grpc.method":"List","grpc.service":"application.ApplicationService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":0.377,"level":"info","msg":"finished unary call with code Unauthenticated","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.method":"Get","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"cluster.SettingsService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /cluster.SettingsService/Get","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"Get","grpc.service":"cluster.SettingsService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":1.133,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.method":"GetUserInfo","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /session.SessionService/GetUserInfo","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"GetUserInfo","grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":0.477,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.method":"GetUserInfo","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /session.SessionService/GetUserInfo","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"GetUserInfo","grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":0.371,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.method":"Get","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"cluster.SettingsService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /cluster.SettingsService/Get","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"Get","grpc.service":"cluster.SettingsService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":1.046,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.method":"GetUserInfo","grpc.request.claims":"null","grpc.request.content":{},"grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","level":"info","msg":"received unary call /session.SessionService/GetUserInfo","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"grpc.code":"OK","grpc.method":"GetUserInfo","grpc.service":"session.SessionService","grpc.start_time":"2020-07-21T21:34:17Z","grpc.time_ms":0.435,"level":"info","msg":"finished unary call with code OK","span.kind":"server","system":"grpc","time":"2020-07-21T21:34:17Z"}
{"level":"info","msg":"Initializing OIDC provider (issuer: https://argocd-host/api/dex)","time":"2020-07-21T21:34:21Z"}

argocd-dex log:

{"level":"info","msg":"config using log level: debug","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"config issuer: https://argocd-host/api/dex","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"config storage: memory","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"config static client: Argo CD","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"config static client: Argo CD CLI","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"config connector: okta","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"config skipping approval screen","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"keys expired, rotating","time":"2020-07-21T21:27:02Z"}
{"level":"info","msg":"keys rotated, next rotation: 2020-07-22 03:27:03.134612551 +0000 UTC","time":"2020-07-21T21:27:03Z"}
{"level":"info","msg":"listening (http/telemetry) on 0.0.0.0:5558","time":"2020-07-21T21:27:03Z"}
{"level":"info","msg":"listening (http) on 0.0.0.0:5556","time":"2020-07-21T21:27:03Z"}
{"level":"info","msg":"listening (grpc) on 0.0.0.0:5557","time":"2020-07-21T21:27:03Z"}