istio: NodePort port incorrectly used in Host header for request routing

Kuberntes: 1.11.1 on minikube v0.28.2 Istio: 1.0.2

I’ve been using Istio on minikube with the ingress gateway using a service of type NodePort.

Most examples in the docs use host * in both their gateways and outward facing virtual services. This works fine, but the moment you want to start trying more complex things and need to actually start routing requests on host name, things fall apart.

Consider the following gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: example
  namespace: example
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - hosts:
    - '*'
    port:
      number: 80
      name: http
      protocol: HTTP

And the following virtual service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: www
  namespace: example
spec:
  hosts:
  - www.example.com
  gateways:
  - example
  http:
  - route:
    - destination:
        host: www

Now suppose that in my local /etc/hosts file, I added the following entry (192.168.99.100 is the IP of my minikube vm):

192.168.99.100 www.example.com

I now get the node port for the ingress gateway:

$ kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}'
31380

Now I try and open http://www.example.com:31380 in my browser.

This does not work. The reason why is clear. Ingress gateway access logs show entries like the following:

[2018-09-18T04:06:14.693Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "172.17.0.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "44659f89-1292-9c6e-8c74-3217db2fd088" "www.example.com:31380" "-"

The host is being, incorrectly, imo, determined to be www.example.com:31380.

I’ll explain why I believe this is incorrect.

By RFC, the Host header sent by a user agent (e.g. browser) may contain trailing port information. In the absence of this, port 80 is assumed for URLs using HTTP as the protocol and 443 is assumed for URLs using HTTPS as the protocol. All browsers that I have checked, omit 80 and 443, respectively for those circumstances, but otherwise do include the port number.

In this case, I can confirm that my browser sends the Host header with the value www.example.com:31380.

This being the case, it seems unsurprising that the gateway sees the Host header as www.example.com:31380– that is, after all, what it was sent… however…

imo, it’s wrong for the gateway to make routing decisions on the basis of this value. Why? Because the request was received on the gateway’s port 80. i.e. The fact that the user agent requested www.example.com:31380 is of no consequence once the gateway has received the request on port 80. imo, it should interpret the Host header as <host:receiving port>-- www.example.com:80 in this case-- and then, by established convention, drop the :80 or :443 if applicable. Here, this would result in www.example.com. This request would be routed correctly.

If there’s any doubt that the behavior I am advocating for here is sensible, please consider that this is how Nginx works.

In Nginx, I have modified the log format as follows:

log_format  main  '$remote_addr !!! $host !!! $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

I’m running this in Kubernetes, with a service of type NodePort– port 30399

With the following entry in my local /etc/hosts:

192.168.99.100 www.foo.com

And a request from my browser to http://www.foo.com:30399

I see access logs such as this:

172.17.0.1 !!! www.foo.com !!! [18/Sep/2018:16:35:28 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "-"

Note that Nginx sensibly disregarded the :30399.

Last, I should acknowledge that there are various spots in the Istio documentation that point out the difficulties of using Istio on minkube and driving traffic through a browser-- where headers aren’t as easily manipulated as they are with curl. The suggested workaround is to only use * as the host in both gateways and virtual services. The problem with this workaround, of course, is that its far too constraining for anyone to attempt anything non-trivial with Istio on minikube…

That difficulty and the need for the very constraining workaround could be remediated by correcting this bug.