linkerd2: Linkerd stable-2.13.4 - linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused

What is the issue?

I have installed Linkerd over Helm (1.12.5) on an EKS cluster and observing the errors listed below…

How can it be reproduced?

Installing Linkerd stable-2.13.4

Logs, error output, etc

linkerd-destination

Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
[     0.002839s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.003837s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.003868s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.003872s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.003875s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.003878s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.003881s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003885s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.004410s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     0.017185s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.110593s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     0.316984s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     0.722792s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     1.223601s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     1.724346s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     2.226078s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     2.727832s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     3.228568s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     3.729321s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     4.231132s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

Workload which has proxy injected

[333255.038731s]  INFO ThreadId(01) linkerd_stack::failfast: Service has recovered
[333256.099452s]  WARN ThreadId(01) outbound:proxy{addr=172.16.64.160:8081}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.129.110.112:8090}: linkerd_reconnect:Failed to connect error=endpoint 10.129.110.112:8090: connect timed out after 1s error.sources=[connect timed out after 1s]
[334074.421535s]  WARN ThreadId(01) outbound:proxy{addr=172.16.0.1:443}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.129.111.71:8090}: linkerd_reconnect: Service failed error=endpoint 10.129.111.71:8090: channel closed error.sources=[channel closed]
[334074.513021s]  WARN ThreadId(01) outbound:proxy{addr=172.16.12.201:6379}:balance{addr=argo-cd-redis-ha-haproxy.argocd.svc.cluster.local:6379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.129.111.71:8086}: linkerd_reconnect: Service failed error=endpoint 10.129.111.71:8086: channel closed error.sources=[channel closed]
[334074.530198s]  WARN ThreadId(01) outbound:proxy{addr=172.16.0.1:443}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.129.111.71:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.129.111.71:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[334074.622297s]  WARN ThreadId(01) outbound:proxy{addr=172.16.12.201:6379}:balance{addr=argo-cd-redis-ha-haproxy.argocd.svc.cluster.local:6379}:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.129.111.71:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.129.111.71:8086: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

output of linkerd check -o short

Status check results are √

Environment

• Kubernetes version: v1.26.6-eks-a5565ad
• Cluster env: EKS
• Host OS: Linux
• Linkerd Version: stable-2.13.4
• CNI: AWS VPC CNI v1.26.2-eksbuild.1

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None