argo-cd: Unable to login against ingress: FATA[0007] rpc error: code = Internal desc = transport: received the unexpected content-type "text/plain; charset=utf-8

Describe the bug After creating an ingress I am unable to login

To Reproduce Steps to reproduce the behavior:

  1. Apply ingress.yaml
  2. Go to ‘argocd.svc.dev.intra’
  3. Enter username and password
    • Password is the one from kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2
  4. Click sign in

Expected behavior I am logged in the same way as it happens when I connect to https://localhost:8080 when doing with port forwarding

Additional context

Login via client:

When try to login via client I get

argocd login argocd.svc.dev.intra
Username: admin
Password:
FATA[0007] rpc error: code = Internal desc = transport: received the unexpected content-type "text/plain; charset=utf-8"

Log of argocd-server:

time="2019-04-09T14:11:26Z" level=info msg="k8s application watch event channel closed"
time="2019-04-09T14:11:26Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:10:24Z" grpc.time_ms=62751.35 span.kind=server system=grpc
time="2019-04-09T14:11:27Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2019-04-09T14:11:27Z" span.kind=server system=grpc
time="2019-04-09T14:11:27Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2019-04-09T14:11:27Z" grpc.time_ms=0.632 span.kind=server system=grpc
time="2019-04-09T14:11:28Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = Invalid username or password" grpc.code=Unauthenticated grpc.method=Create grpc.service=session.SessionService grpc.start_time="2019-04-09T14:11:28Z" grpc.time_ms=74.912 span.kind=server system=grpc
time="2019-04-09T14:11:32Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"iat\":1553886824,\"iss\":\"argocd\",\"nbf\":1553886824,\"sub\":\"admin\"}" grpc.request.content= grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:11:32Z" span.kind=server system=grpc
time="2019-04-09T14:11:32Z" level=info msg="client watch grpc context closed"
time="2019-04-09T14:11:32Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:11:32Z" grpc.time_ms=16.728 span.kind=server system=grpc
time="2019-04-09T14:11:32Z" level=info msg="k8s application watch event channel closed"
time="2019-04-09T14:11:32Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"iat\":1553886824,\"iss\":\"argocd\",\"nbf\":1553886824,\"sub\":\"admin\"}" grpc.request.content= grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:11:32Z" span.kind=server system=grpc

ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: argocd.svc.dev.intra
    http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: 443
  tls:
  - hosts:
    - argocd.svc.dev.intra
    secretName: wildcard-ingress-cert

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (1 by maintainers)

Most upvoted comments

I ran into this issue even after setting up two separate ingresses for http and grpc as describe in the docs.

argocd login argocd.xxxxx.com --grpc-web was the only thing that worked for me, per @Sicaine

I have the same issue with traefik but it got solved by adding --grpc-web: “Enables gRPC-web protocol. Useful if Argo CD server is behind proxy which does not support HTTP2”

no answer fix my issue … i am getting this error:

argocd login cd.devops.company.com --grpc-web
WARNING: server is not configured with TLS. Proceed (y/n)? y
Username: admin
Password:
FATA[0022] rpc error: code = Unknown desc = Post "http://cd.devops.company.com:443/session.SessionService/Create": EOF

Can you notice with me that the log shows http://cd.devops.company.com:443 while i expect to be: https://cd.devops.company.com:443 (https not http).

While argocd CLI use http ?

Also i am in this situation:

  1. It works fine in the browser (https://cd.devops.company.com)
  2. the request flow is : client --(443)–> corporate loadbalancer --(80)–> ingress-controller --(app-port)–> argocd-service —> argocd-pod …

The error: received the unexpected content-type "text/plain may mean you are hitting the default ingress backend, which means ssl-passthrough may not be taking effect. Note that nginx.ingress.kubernetes.io/ssl-passthrough requires an additional --enable-ssl-passthrough option to nginx-ingress.

The error: received the unexpected content-type "text/plain may mean you are hitting the default ingress backend, which means ssl-passthrough may not be taking effect. Note that nginx.ingress.kubernetes.io/ssl-passthrough requires an additional --enable-ssl-passthrough option to nginx-ingress.

If you use the helm chart for nginx-ingress, then just add this as an extraArg to the controller in values.yaml, and run helm upgrade nginx-ingress.

extraArgs:
    enable-ssl-passthrough: ""