linkerd2: Proxy fails to route to updated pod deployment

Bug Report

What is the issue?

After running a deploy via Spinnaker, we were alerted that traffic flow to the pod was completely cut off, returning a 502 Bad Gateway for all inbound HTTP traffic.

How can it be reproduced?

As of now, we’ve been unable to reproduce it. We ran linkerd uninject and redeployed the now-uninjected service. After re-injecting, the behavior was no longer present.

Logs, error output, etc

The log output is repeated based on the block below - this was taken from one of the proxies running alongside NGINX:

nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.174492s] linkerd2_proxy::app::profiles error fetching profile for imeimsisdnrelay.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.174643s] linkerd2_proxy::app::profiles error fetching profile for cricalllog.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.175601s] linkerd2_proxy::app::profiles error fetching profile for criincomingcall.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.175751s] linkerd2_proxy::app::profiles error fetching profile for callterminationreceiver.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.175832s] linkerd2_proxy::app::profiles error fetching profile for dhcadmin.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.175904s] linkerd2_proxy::app::profiles error fetching profile for accountreceiver.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.175970s] linkerd2_proxy::app::profiles error fetching profile for nginx-auth-service.infra.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.176054s] linkerd2_proxy::app::profiles error fetching profile for dcmauthreceiver.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.176119s] linkerd2_proxy::app::profiles error fetching profile for opsdashboard.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.176189s] linkerd2_proxy::app::profiles error fetching profile for teapi.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.176262s] linkerd2_proxy::app::profiles error fetching profile for vehiclelocator.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.181130s] linkerd2_proxy::app::profiles error fetching profile for historyapi.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.181144s] linkerd2_proxy::app::profiles error fetching profile for web-svc.emojivoto.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.181151s] linkerd2_proxy::app::profiles error fetching profile for provapi.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.181157s] linkerd2_proxy::app::profiles error fetching profile for linkerd-web.linkerd.svc.cluster.local:8084: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.181162s] linkerd2_proxy::app::profiles error fetching profile for ikura.tsp.svc.cluster.local:8000: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.181168s] linkerd2_proxy::app::profiles error fetching profile for linkerd-web.linkerd.svc.cluster.local:8084: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484416.181190s] linkerd2_proxy::app::profiles error fetching profile for infraapi.tsp.svc.cluster.local:80: Status { code: Unknown, message: "the request could not be dispatched in a timely fashion" }
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.172176s] proxy={bg=resolver addr=ikura.tsp.svc.cluster.local:8000} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.172558s] proxy={bg=resolver addr=infraapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.172714s] proxy={bg=resolver addr=imeimsisdnrelay.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.172860s] proxy={bg=resolver addr=cricalllog.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.172991s] proxy={bg=resolver addr=criincomingcall.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.173097s] proxy={bg=resolver addr=callterminationreceiver.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.173209s] proxy={bg=resolver addr=historyapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.173300s] proxy={bg=resolver addr=dhcadmin.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.173503s] proxy={bg=resolver addr=nginx-auth-service.infra.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.173708s] proxy={bg=resolver addr=vehicletracker.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.173889s] proxy={bg=resolver addr=opsdashboard.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.173980s] proxy={bg=resolver addr=teapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.174115s] proxy={bg=resolver addr=vehiclelocator.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.175316s] proxy={bg=resolver addr=web-svc.emojivoto.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.175416s] proxy={bg=resolver addr=vlapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.175518s] proxy={bg=resolver addr=accountreceiver.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.175959s] proxy={bg=resolver addr=provapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.176206s] proxy={bg=resolver addr=ikura.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.176300s] proxy={bg=resolver addr=linkerd-web.linkerd.svc.cluster.local:8084} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.176399s] proxy={bg=resolver addr=linkerd-web.linkerd.svc.cluster.local:8084} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484418.176492s] proxy={bg=resolver addr=dcmauthreceiver.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.174552s] proxy={bg=resolver addr=ikura.tsp.svc.cluster.local:8000} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.174730s] proxy={bg=resolver addr=historyapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.174822s] proxy={bg=resolver addr=dhcadmin.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.174984s] proxy={bg=resolver addr=infraapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175074s] proxy={bg=resolver addr=imeimsisdnrelay.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175182s] proxy={bg=resolver addr=cricalllog.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175262s] proxy={bg=resolver addr=criincomingcall.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175335s] proxy={bg=resolver addr=callterminationreceiver.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175429s] proxy={bg=resolver addr=nginx-auth-service.infra.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175452s] proxy={bg=resolver addr=vehicletracker.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175667s] proxy={bg=resolver addr=opsdashboard.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175686s] proxy={bg=resolver addr=teapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175760s] proxy={bg=resolver addr=vehiclelocator.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175844s] proxy={bg=resolver addr=web-svc.emojivoto.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.175920s] proxy={bg=resolver addr=vlapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.177911s] proxy={bg=resolver addr=ikura.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.177939s] proxy={bg=resolver addr=linkerd-web.linkerd.svc.cluster.local:8084} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.178072s] proxy={bg=resolver addr=linkerd-web.linkerd.svc.cluster.local:8084} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.178092s] proxy={bg=resolver addr=dcmauthreceiver.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.178164s] proxy={bg=resolver addr=accountreceiver.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"
nginx-ingress-controller-6f696ddb9f-nz58s linkerd-proxy WARN [484423.178238s] proxy={bg=resolver addr=provapi.tsp.svc.cluster.local:80} linkerd2_proxy::control::destination::resolution Destination.Get stream error: grpc-status: Unknown, grpc-message: "the request could not be dispatched in a timely fashion"

linkerd check output

❯ linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ control plane namespace exists
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ no invalid service profiles

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

Status check results are √

Environment

• Kubernetes Version: 1.12.7
• Cluster Environment: AKS
• Host OS: Ubuntu
• Linkerd version: 2.3.2

Possible solution

We re-injected the problem pods before opening this issue and haven’t experienced any of the failures seen in the past. We’re monitoring to see if anything regresses, but as of now all workloads are stable - given that, it seems like an un-inject followed by a re-inject seems to work.

Additional context

Output from kubectl exec -n linkerd -it -c linkerd-proxy linkerd-identity-7dcb854d79-nk76w -- dig linkerd-identity.linkerd.svc.cluster.local:

❯ kubectl exec -n linkerd -it -c linkerd-proxy linkerd-identity-d69b76885-crxn4 -- dig linkerd-identity.linkerd.svc.cluster.local

; <<>> DiG 9.10.3-P4-Debian <<>> linkerd-identity.linkerd.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12462
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;linkerd-identity.linkerd.svc.cluster.local. IN A

;; ANSWER SECTION:
linkerd-identity.linkerd.svc.cluster.local. 5 IN A 10.250.162.77

;; Query time: 0 msec
;; SERVER: 10.250.0.10#53(10.250.0.10)
;; WHEN: Wed Jun 12 14:55:31 UTC 2019
;; MSG SIZE  rcvd: 118