linkerd2: proxy: connection refused when service is on a non standard port

Bug Report

What is the issue?

We have created an ingress for prometheus (service port 9090) and alertmanager (service port 9093). When browsing to the services, the ingress controller’s logs the usual access log entry (Everything looks correct) but the proxy logs an error.

How can it be reproduced?

Logs, error output, etc

2019-01-21T08:33:52.761022118Z ERR! proxy={server=out listen=127.0.0.1:4140 remote=10.1.65.239:60066} linkerd2_proxy::proxy::http::router service error: an error occurred trying to connect: Connection refused (os error 111) (address: 10.1.64.38:80)

2019-01-21T08:33:52.761317114Z {
  "time": "2019-01-21T08:33:52+00:00",
  "request_id": "283f286b8b8029482e06e3c09dc1263f",
  "source_ip": "127.0.0.1",
  "remote_addr": "127.0.0.1",
  "x-forward-for": "127.0.0.1",
  "remote_user": "",
  "request_proto": "HTTP/2.0",
  "method": "GET",
  "vhost": "prometheus.[internal.domain]",
  "path": "/",
  "request_query": "",
  "request_length": 13,
  "request_duration": "0.001",
  "status": "500",
  "upstream_status": "500",
  "response_body_bytes": "0",
  "upstream_name": "monitoring-prometheus-prometheus-9090",
  "upstream_ip": "10.1.64.38:9090",
  "upstream_response_time": "0.004",
  "upstream_response_length": "0",
  "http_referrer": "",
  "http_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0",
  "ingress_namespace": "monitoring",
  "ingress_name": "prometheus-prometheus", 
  "service_name": "prometheus-prometheus", 
  "service_port": "9090"
}

as you can see, nginx knows the request should be going to port 9090, but linkerd is trying to connect to port 80. If I uninject the ingress controller, it works fine

linkerd check output

kubernetes-api                                                                                                                                                                                                                                                                                                                  --------------                                                                                                                                                                                                                                                                                                                  ✔ can initialize the client                                                                                                                                                                                                                                                                                                     ✔ can query the Kubernetes API                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  kubernetes-version                                                                                                                                                                                                                                                                                                              ------------------                                                                                                                                                                                                                                                                                                              ✔ is running the minimum Kubernetes API version                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 linkerd-existence                                                                                                                                                                                                                                                                                                               -----------------                                                                                                                                                                                                                                                                                                               ✔ control plane namespace exists                                                                                                                                                                                                                                                                                                ✔ controller pod is running                                                                                                                                                                                                                                                                                                     ✔ can initialize the client                                                                                                                                                                                                                                                                                                     ✔ can query the control plane API                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               linkerd-api                                                                                                                                                                                                                                                                                                                     -----------                                                                                                                                                                                                                                                                                                                     ✔ control plane pods are ready                                                                                                                                                                                                                                                                                                  ✔ can query the control plane API                                                                                                                                                                                                                                                                                               ✔ [kubernetes] control plane can talk to Kubernetes                                                                                                                                                                                                                                                                             ✔ [prometheus] control plane can talk to Prometheus                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             linkerd-service-profile                                                                                                                                                                                                                                                                                                         -----------------------                                                                                                                                                                                                                                                                                                         ✔ no invalid service profiles                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   linkerd-version                                                                                                                                                                                                                                                                                                                 ---------------                                                                                                                                                                                                                                                                                                                 ✔ can determine the latest version                                                                                                                                                                                                                                                                                              ✔ cli is up-to-date                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             control-plane-version                                                                                                                                                                                                                                                                                                           ---------------------                                                                                                                                                                                                                                                                                                           ✔ control plane is up-to-date                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   Status check results are ✔ 

Environment

• Kubernetes Version: 1.13.2
• Cluster Environment: aks-engine
• Host OS: aks
• Linkerd version: 1.19.2

Possible solution

If I update the application’s service to register a port 80, it works. however, after a while (unknown how long), linkerd starts logging connection timeout after 300ms to the service

Additional context