argo-cd: Dex SSO failing with "http: server gave HTTP response to HTTPS client"
Checklist:
- I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I’ve included steps to reproduce the bug.
- I’ve pasted the output of
argocd version.
Describe the bug
Receiving following error message when trying to login with SSO to argocd:
Failed to query provider "https://<server>/api/dex": Get "https://argocd-dex-server:5556/api/dex/.well-known/openid-configuration": http: server gave HTTP response to HTTPS client
I’m wondering if it could be related to any of the following recent changes:
- https://github.com/argoproj/argo-cd/releases/tag/v2.4.5
- https://github.com/argoproj/argo-cd/security/advisories/GHSA-7943-82jg-wmw5
- https://github.com/argoproj/argo-cd/issues/9424
We are using the bundled dex instance.
I’ve tried changing the oidc.tls.insecure.skip.verify setting to true as suggested but haven’t seen any improvement. I’m considering pinning all the container images to 2.4.4 in the cluster but wanted to see if anyone else was seeing this or had thoughts first.
To Reproduce
We see this error whenever we click on the SSO login button in the UI.
Expected behavior
Login should occur.
Version
We do not have the default admin login turned on and because our CLI login uses SSO, I’m not able to get the actual output.
However our manifests are configured to use the latest version of argocd and so I would think we are on 2.4.5 or 2.4.6.
Updated version info after getting SSO working again with the changes below
$ argocd version --grpc-web
argocd: v2.1.6+a346cf9.dirty
BuildDate: 2021-11-01T02:06:48Z
GitCommit: a346cf933e10d872eae26bff8e58c5e7ac40db25
GitTreeState: dirty
GoVersion: go1.17.2
Compiler: gc
Platform: darwin/amd64
argocd-server: v2.4.0+7d31d61
BuildDate: 2022-07-14T21:08:57Z
GitCommit: 7d31d612ef2309a4d6dfef3a47e565af4742586f
GitTreeState: clean
GoVersion: go1.18.4
Compiler: gc
Platform: linux/amd64
Kustomize Version: v4.5.5 2022-05-20T20:25:40Z
Helm Version: v3.9.0+g7ceeda6
Kubectl Version: v0.24.2
Jsonnet Version: v0.18.0
Logs
Unable to find any relevant logs.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 3
- Comments: 18
If you are using Argo CD <v2.5.0, then @tonedefdev’s solution above is acceptable. But please make a note that when you upgrade to >=v2.5.0, you should remove these settings to take advantage of HTTPS communication between the API server and Dex.
If you are using Argo CD >=v2.5.0, please do not disable TLS as a fix for this error message. TLS support was added because it is a very important security mechanism for a very sensitive communication channel.
This error message occurs for one of two reasons. Either:
"http: server gave HTTP response to HTTPS client": The API server expected to communicate via HTTPS, but Dex expected to communicate via HTTP."http: server gave HTTPS response to HTTP client": The API server expected to communicate via HTTP, but Dex expected to communicate via HTTPS.This configuration mismatch is a solveable problem, and it should be solved by identifying the mismatch rather than hacked around by making things less secure.
A common configuration mismatch is when folks manually set
global.image.tagin the argo-helm argo-cd chart. This may cause the Argo CD image to not match the manifests.There are a thousand other possible ways to produce a mismatch, so I can’t enumerate the solutions here. @tonedefdev found a valid solution was manually disabling TLS. This is valid, because their version of Argo CD does not support TLS communication between the API server and Dex. But if they were using 2.5.0+, they would need to dig deeper to find the config mismatch.
@crenshaw-dev sorry, the formatting got messed up for some reason.
Basically the redirect does not result in the argocd ui being loaded. Instead we see a page that just shows the jwt token from our IdP.
However, the redirect does add the correct cookie, so if we rewrite the url in the browser to being the argocd homepage we are able to log in.
Yeah, that makes no sense to me, because the OIDC client should be using whatever schema is set in
server.dex.server.Apologies for the slow response, coming back to this tomorrow.