linkerd2: Connection closed error=proxy admin server is HTTP-only

Bug Report

What is the issue?

The communication between prometheus and the proxy should be using HTTPS but that is not the case.

How can it be reproduced?

Upgrade from version stable-2.9.1 to version edge-21.1.4

Initially, it was installed using:

linkerd install \
  --identity-trust-anchors-file ca.crt \
  --identity-issuer-certificate-file issuer.crt \
  --identity-issuer-key-file issuer.key \
  | kubectl apply -f -

For upgrading I did:

linkerdEdge upgrade | tee linkerd/base/linkerd.yaml
linkerdEdge viz install | tee linkerd/base/linkerd-viz.yaml

and argo replaced the old YAML files with the new ones.

Logs, error output, etc

INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=<prometheus pod ip>:35390 target.addr=<pgbouncer pod ip>:4191}: linkerd_app_core::serve: Connection closed error=proxy admin server is HTTP-only

`linkerd check` output

I have already rolled it back.

Environment

Kubernetes Version: 1.16
Kustomize Version: v3.8.1 (argo CD)
Cluster Environment: (AWS)
Linkerd version: edge-21.1.4

Possible solution

Maybe the same flags (for identity) should be available for viz

Additional context

I was using opaque-ports for pgbouncer (5432) and I have rolled it back already

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 19 (12 by maintainers)

Commits related to this issue

admin: Handle connections that fail protocol detection The HTTP detect module may fail to detect an HTTP protocol if the first read does not return at least 14 bytes. This has caused spurious failure... — committed to linkerd/linkerd2-proxy by olix0r 3 years ago
admin: Handle connections that fail protocol detection (#960) The HTTP detect module may fail to detect an HTTP protocol if the first read does not return at least 14 bytes. This has caused spurious... — committed to linkerd/linkerd2-proxy by olix0r 3 years ago

Most upvoted comments

I investigated one case, I didn’t have raw log available anymore, but I was using logs in kibana. For example when we had this 0x16 in our logs, like 760 messages per second, in the same second there were 76 following messages in proxy-log:

{"timestamp":"...","level":"INFO","fields":{"message":"Connection closed","error":"request timed out"},"target":"linkerd_app_core::serve","spans":[{"name": "inbound"},{"client_addr": "10.1.21.79:33178","target_addr": "10.1.4.38:8080","name": "accept"}],"threadId":"ThreadId(1)"}

and 8 following messages:

{"timestamp":"...","level":"INFO","fields":{"message":"Connection closed","error":"proxy admin server is HTTP-only"},"target":"linkerd_app_core::serve","spans":[{"name": "daemon"},{"listen_addr": "0.0.0.0:4191","name": "admin"},{"client_addr": "10.1.26.203:45330","target_addr": "10.1.4.38:4191","name": "accept"}],"threadId":"ThreadId(2)"}

10.1.21.79 belonged to traefik pod and 10.1.4.38 to linkerd-viz.prometheus. I checked also traefik pod proxy log, there was silence at that time.

tarvip on Apr 13, 2021

Just wanted to add our experience to this: clean 2.10 install on AWS EKS. Getting a lot of errors on the linkerd-proxy logs for 1 particular type of service we have (it listens on 0.0.0.0): [130883.786625s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=192.168.150.45:51202 target.addr=192.168.175.75:4191}: linkerd_app_core::serve: Connection closed error=proxy admin server is HTTP-only [130884.290767s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=192.168.150.45:51296 target.addr=192.168.175.75:4191}: linkerd_app_core::serve: Connection closed error=proxy admin server is HTTP-only [130884.793236s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=192.168.150.45:51370 target.addr=192.168.175.75:4191}: linkerd_app_core::serve: Connection closed error=proxy admin server is HTTP-only [130885.298089s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}:accept{client.addr=192.168.150.45:51448 target.addr=192.168.175.75:4191}: linkerd_app_core::serve: Connection closed error=proxy admin server is HTTP-only

The client address here is linkerd viz prometheus . I ended up uninstalling linkerd viz since it was spamming my logs

virenrshah on Mar 24, 2021

Thanks @possani! We’ll look into this and let you know if we need any further information.

adleong on Feb 10, 2021