oauth2-proxy: Using some extra args with Azure results in error 400 and 500

Edit: When using the following extraArgs with provider: azure, successful authentication will lead to error 400 returned from login.live.com and result in error 500 shown to the user:

• cookie-refresh
• pass-authorization-header: true
• set-authorization-header: true
• pass-access-token: true

Original text:

When going to the application URL (e.g. app.private.example.com), redirect to oauth2 proxy (at auth.private.example.com) and then to authentication provider (azure) happens fine. I put in my credentials and log in, it redirects me back to oauth2 proxy and then to application URL. However, I never get to the application as error 500 is shown. Subsequent attempts to go to the application domain go straight to error 500.

Been banging my head on this for the past 12 hours and would appreciate any help.

Expected Behavior

Expected to see the application

Current Behavior

See error 500

During subsequent attempts (after having logged in) I can see in nginx ingress controller log the following:

2020/03/26 15:24:33 [error] 1864#1864: *564714 auth request unexpected status: 400 while sending to client, client: 10.244.0.1, server: app.private.example.com.com, request: "GET / HTTP/2.0", host: "app.private.example.com.com", referrer: "https://login.live.com/"
10.244.0.1 - - [26/Mar/2020:15:24:33 +0000] "GET / HTTP/2.0" 400 0 "https://login.live.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 0 0.130 [kube-system-kubernetes-dashboard-80] [] 51.140.27.8:443 0 0.004 400 4f6d118409e294abff873e46eb91c583
10.244.0.1 - - [26/Mar/2020:15:24:33 +0000] "GET / HTTP/2.0" 500 579 "https://login.live.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 7764 0.130 [kube-system-kubernetes-dashboard-80] [] - - - - 4f6d118409e294abff873e46eb91c583
10.244.0.1 - - [26/Mar/2020:15:24:33 +0000] "GET /oauth2/auth HTTP/1.1" 400 635 "https://login.live.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 827 0.000 [] [] - - - - c56c2e5b45b728ff5e41cbf3373e418a
2020/03/26 15:24:33 [error] 1864#1864: *564714 auth request unexpected status: 400 while sending to client, client: 10.244.0.1, server: app.private.example.com.com, request: "GET /favicon.ico HTTP/2.0", host: "app.private.example.com.com", referrer: "https://app.private.example.com.com/"
10.244.0.1 - - [26/Mar/2020:15:24:33 +0000] "GET /favicon.ico HTTP/2.0" 400 0 "https://app.private.example.com.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 0 0.003 [kube-system-kubernetes-dashboard-80] [] 51.140.27.8:443 0 0.004 400 90f74ae4624554e74b4cb0802e586130
10.244.0.1 - - [26/Mar/2020:15:24:33 +0000] "GET /favicon.ico HTTP/2.0" 500 579 "https://app.private.example.com.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 7698 0.003 [kube-system-kubernetes-dashboard-80] [] - - - - 90f74ae4624554e74b4cb0802e586130
10.244.0.1 - - [26/Mar/2020:15:24:33 +0000] "GET /oauth2/auth HTTP/1.1" 400 635 "https://app.private.example.com.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 740 0.000 [] [] - - - - 7f27222dcedfa1f275105a787fd5e882

Logs of oauth2 proxy show nothing except kube-probe pings.

I have checked and I can see the cookies _oauth2_proxy_0, _oauth2_proxy_1 and _oauth2_proxy_2 that appear to be being set correctly.

Steps to Reproduce (configurations)

Dashboard ingress

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-dashboard
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/auth-url: "https://auth.private.example.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://auth.private.example.com/oauth2/start?rd=https%3A%2F%2F$host$request_uri"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "64k"

spec:
  tls:
  - hosts:
    - app.private.example.com
    secretName: tls-secret-kubernetes-dashboard
  rules:
  - host: app.private.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

---

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: tls-secret-kubernetes-dashboard
  namespace: kube-system
spec:
  secretName: tls-secret-kubernetes-dashboard
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName:  app.private.example.com
  dnsNames:
  -  app.private.example.com
  renewBefore: 720h
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - app.private.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

oauth2 proxy values.yaml

config:
  existingSecret: oauth2-proxy-creds

extraArgs:
  provider: azure
  azure-tenant: <redacted>
  whitelist-domain: .private.example.com
  cookie-domain: .private.example.com
  cookie-secure: true
  redirect-url: "https://auth.private.example.com/oauth2/callback" # If we don't specify this, redirect URL sent to provider may use HTTP rather than HTTPS, which may be picked as a mismatch (e.g. in Azure)
  scope: openid offline_access User.Read
  cookie-refresh: 24h
  cookie-samesite: none # Chrome won't play with anything except none with HTTPS

extraEnv:
- name: OAUTH2_PROXY_AZURE_TENANT
  value : <redacted>

authenticatedEmailsFile:
  enabled: true
  restricted_access: |-
    support@example.com

ingress:
  enabled: true
  path: /
  hosts:
    - auth.private.example.com
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-buffer-size: "64k"
  tls:
    - secretName: oauth2-proxy-https-cert
      hosts:
        - auth.private.example.com

Context

I am trying to set up external authentication with oauth2 proxy, kubernetes dashboard, ingress and letsencrypt. For now, I only want authentication when logged in with support@example.com.

Possible Solution

I’m sure this has to be something in configuration.

I have tried to add the following bits to extraArgs for oauth2 proxy:

  set-authorization-header: true
  reverse-proxy: true
  pass-authorization-header: true
  pass-access-token: true
  pass-basic-auth: true
  pass-host-header: true
  pass-user-headers: true
  set-xauthrequest: true

and (at the same time) the following to Dashboard ingress annotations:

    nginx.ingress.kubernetes.io/auth-response-headers: Authorization
    nginx.ingress.kubernetes.io/configuration-snippet: |
      auth_request_set $name_upstream_1 $upstream_cookie__oauth2_proxy_1;

      access_by_lua_block {
        if ngx.var.name_upstream_1 ~= "" then
          ngx.header["Set-Cookie"] = "name_1=" .. ngx.var.name_upstream_1 .. ngx.var.auth_cookie:match("(; .*)")
        end
      }

But this had no effect.

I am really at a loss. It looks the error 400 is produced by oauth2 proxy container, but there is nothing in its logs.

Your Environment

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.7", GitCommit:"20c697423de11d4d5d08c6832a513217ca11a6aa", GitTreeState:"clean", BuildDate:"2020-02-21T10:47:45Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

nginx-ingress 0.30.0

• Version used: oauth2-proxy 5.0.0