istio: 404 if VirtualService.spec.hosts is set to FQDN

Describe the bug I tried to see how the basic examples in the official istio docs work. Everything works as prescribed but only as long as VirtualService.spec.hosts == "*". When I set it to fully qualified domain name (FQDN) envoy is giving me 404.

Expected behavior Get response from the configured service.

Steps to reproduce the bug

Set up local hosts file:

# echo 'xxx.yyy.122.33 bookinfo.example.com' > /etc/hosts

Create custom Namespace with enabled automatic istio sidecar injection:

---
apiVersion: v1
kind: Namespace
metadata:
  name: bookinfo
  labels:
    istio-injection: enabled
---

Deploy bookinfo components in that Namespace:

$ kubectl create -f https://raw.githubusercontent.com/istio/istio/release-1.0/samples/bookinfo/platform/kube/bookinfo.yaml
service "details" created
deployment.extensions "details-v1" created
service "ratings" created
deployment.extensions "ratings-v1" created
service "reviews" created
deployment.extensions "reviews-v1" created
deployment.extensions "reviews-v2" created
deployment.extensions "reviews-v3" created
service "productpage" created
deployment.extensions "productpage-v1" created

$ kubectl get po
NAME                            READY     STATUS    RESTARTS   AGE
details-v1-876bf485f-2d4sr      2/2       Running   0          16s
productpage-v1-8d69b45c-p7kmn   2/2       Running   0          15s
ratings-v1-7c9949d479-k55cb     2/2       Running   0          15s
reviews-v1-85b7d84c56-xgl8m     2/2       Running   0          15s
reviews-v2-cbd94c99b-79lx2      2/2       Running   0          15s
reviews-v3-748456d47b-kplrb     2/2       Running   0          15s

$ kubectl get svc
NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
details       ClusterIP   10.30.224.45   <none>        9080/TCP   20s
productpage   ClusterIP   10.30.111.5    <none>        9080/TCP   19s
ratings       ClusterIP   10.30.50.27    <none>        9080/TCP   20s
reviews       ClusterIP   10.30.93.28    <none>        9080/TCP   19s

Declare Gateway and VirtualService:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway

metadata:
  namespace: bookinfo
  name: bookinfo-gw

spec:

  selector:
    istio: ingressgateway

  servers:

  - port:
      number: 80
      protocol: HTTP
      name: http
    hosts:
    - bookinfo.example.com
---


---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService

metadata:
  namespace: bookinfo
  name: bookinfo-virt-svc

spec:

  gateways:
  - bookinfo-gw

  hosts:
  - "*"

  http:

  - route:
    - destination:
        host: productpage.bookinfo.svc.cluster.local
        port:
          number: 9080
---

Deploy Gateway and VirtualService:

$ kubectl create -f istio-test.yaml 
gateway.networking.istio.io "bookinfo-gw" created
virtualservice.networking.istio.io "bookinfo-virt-svc" created

Test connectivity:

$ curl -LsD- -o /dev/null bookinfo.example.com:31380
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 1836
server: envoy
date: Mon, 18 Feb 2019 17:19:32 GMT
x-envoy-upstream-service-time: 25

Up to this point everything has been configured pretty much according to the book.

Now change VirtualService.spec.hosts to bookinfo.example.com:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService

metadata:
  namespace: bookinfo
  name: bookinfo-virt-svc

spec:

  gateways:
  - bookinfo-gw

  hosts:
  - bookinfo.example.com

  http:

  - route:
    - destination:
        host: productpage.bookinfo.svc.cluster.local
        port:
          number: 9080
---

Deploy it:

$ kubectl replace -f istio-test.yaml 
virtualservice.networking.istio.io "bookinfo-virt-svc" replaced

Test the connectivity:

$ curl -LsD- -o /dev/null bookinfo.example.com:31380
HTTP/1.1 404 Not Found
date: Mon, 18 Feb 2019 17:23:23 GMT
server: envoy
content-length: 0

Some debugging info:

$ istioctl proxy-config routes istio-ingressgateway-78c6d8b8d7-pdtb8 -n istio-system -o json
[
    {
        "name": "http.80",
        "virtualHosts": [
            {
                "name": "bookinfo.example.com:80",
                "domains": [
                    "bookinfo.example.com",
                    "bookinfo.example.com:80"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/"
                        },
                        "route": {
                            "cluster": "outbound|9080||productpage.bookinfo.svc.cluster.local",
                            "timeout": "0.000s",
                            "maxGrpcTimeout": "0.000s"
                        },
                        "decorator": {
                            "operation": "productpage.bookinfo.svc.cluster.local:9080/*"
                        },
                        "perFilterConfig": {
                            "mixer": {
                                "forward_attributes": {
                                    "attributes": {
                                        "destination.service": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.host": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "string_value": "productpage"
                                        },
                                        "destination.service.namespace": {
                                            "string_value": "bookinfo"
                                        },
                                        "destination.service.uid": {
                                            "string_value": "istio://bookinfo/services/productpage"
                                        }
                                    }
                                },
                                "mixer_attributes": {
                                    "attributes": {
                                        "destination.service": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.host": {
                                            "string_value": "productpage.bookinfo.svc.cluster.local"
                                        },
                                        "destination.service.name": {
                                            "string_value": "productpage"
                                        },
                                        "destination.service.namespace": {
                                            "string_value": "bookinfo"
                                        },
                                        "destination.service.uid": {
                                            "string_value": "istio://bookinfo/services/productpage"
                                        }
                                    }
                                }
                            }
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/stats/prometheus"
                        },
                        "route": {
                            "cluster": "prometheus_stats"
                        }
                    }
                ]
            }
        ]
    }
]

Version

$ kubectl version --short
Client Version: v1.12.4
Server Version: v1.12.4

$ istioctl version
Version: 1.0.5
GitRevision: c1707e45e71c75d74bf3a5dec8c7086f32f32fad
User: root@6f6ea1061f2b
Hub: docker.io/istio
GolangVersion: go1.10.4
BuildStatus: Clean

Installation

#! /bin/bash --

helm template install/kubernetes/helm/istio\
 --name istio\
 --namespace istio-system\
 --set kiali.enabled=true\
 --set grafana.enabled=true\
 --set tracing.enabled=true\
 --set servicegraph.enabled=true |\
 sed 's/type: LoadBalancer/type: NodePort/'

Environment Bare-metal server

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 20 (7 by maintainers)

Most upvoted comments

@narun4sk try to use authority match, instead of using host headers in your VirtualService

- match:
    - authority:
         exact: "bookinfo.example.com:31380"
+7
erezzarum on Feb 19, 2019

@zhongfox this stackoverflow question answers that: https://stackoverflow.com/questions/2142910/whats-the-difference-between-uri-host-and-uri-authority.

Authority -> host + port Host -> host

It’s also important to note that this question isn’t asking about FQDNs 😃. Any FQDN is a domain terminated with . indicating the domain root. Otherwise it’s a relative domain. This question is about domain + port matching.

+2
Freyert on Oct 11, 2019

I have checked this issue on the istio version (1.4.6, 1.4.7, 1.5.2) and it’s still there. So if you are using a custom ports to connect to service - envoy will receive header as domain:port and it will cause a problem that envoy can not match expected domain name and received domain:port. As a temporary workaround - you can create a nginx proxy with config like:

## tcp LB  and SSL passthrough for backend ##
stream {
    upstream apigee {
        server K8S-NODE-IP:NODEPORT;
    }

    log_format basic '$remote_addr [$time_local] '
                 '$protocol $status $bytes_sent $bytes_received '
                 '$session_time "$upstream_addr" '
                 '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

    access_log /var/log/nginx/apigee_access.log basic;
    error_log  /var/log/nginx/apigee_error.log;

    server {
        listen 443;
        proxy_pass apigee;
        proxy_next_upstream on;
    }
}

http://nginx.org/en/docs/stream/ngx_stream_core_module.html

curl -v http://IP:PORT/hello
*   Trying IP...
* TCP_NODELAY set
* Connected to IP (IP) port NODEPORT (#0)
> GET /hello HTTP/1.1
> Host: IP:NODEPORT
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 60
< server: istio-envoy
< date: Tue, 28 Apr 2020 10:41:36 GMT
< x-envoy-upstream-service-time: 176
<
Hello version: v1, instance: helloworld-v1-578dd69f69-6srxc
* Connection #0 to host IP left intact
* Closing connection 0

curl -v -H "Host:helloworld.test.com" http://helloworld.test.com:NODEPORT/hello
*   Trying IP...
* TCP_NODELAY set
* Connected to helloworld.test.com (IP) port NODEPORT (#0)
> GET /hello HTTP/1.1
> Host:helloworld.test.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 60
< server: istio-envoy
< date: Tue, 28 Apr 2020 10:45:10 GMT
< x-envoy-upstream-service-time: 165
<
Hello version: v2, instance: helloworld-v2-776f74c475-q7p7q
* Connection #0 to host helloworld.test.com left intact
* Closing connection 0

istioctl version
client version: 1.5.0
control plane version: 1.5.2
data plane version: 1.5.2 (5 proxies)

Istio 1.4.7:
envoy --version envoy  version: f777dd13e28edc4ff394e98cd82c82ca4a2bcc71/1.12.0/Clean/RELEASE/BoringSSL

Istio 1.5.2:
envoy --version
envoy  version: 54fd05bc1e3e59a155651e5c84e4138d849e8d58/1.13.1-dev/Clean/RELEASE/BoringSSL

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: helloworld-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    #- "*"
    - "helloworld.test.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld
spec:
  hosts:
  - "helloworld.test.com"
  #- "*"
  gateways:
  - helloworld-gateway
  http:
  - match:
    - uri:
        exact: /hello
    route:
    - destination:
        host: helloworld
        port:
          number: 5000
+1
skolenkin on Apr 28, 2020

Another workaround might be to use the same port as your NodePort (31380) in your Istio ingressgateway service port and target port and then setup the Gateway with port 31380.

This will ensure that Host:31380 is accepted by the ingress proxy as the Gateway port will be added to the host declared in the VirtualService resource.

+1
nrjpoddar on Feb 20, 2019

I also tried the following configuration but it too is giving me 404:

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService

metadata:
  name: bookinfo-virt-svc

spec:

  gateways:
  - bookinfo-gw

  hosts:
  - "*"

  http:

  - match:
    - headers:
        host:
          exact: "bookinfo.example.com:31380"
    route:
    - destination:
        host: productpage.bookinfo.svc.cluster.local
        port:
          number: 9080
---

$ curl -LsD- -o /dev/null bookinfo.example.com:31380
HTTP/1.1 404 Not Found
date: Tue, 19 Feb 2019 20:38:07 GMT
server: envoy
content-length: 0

$ kubectl logs -n istio-system istio-ingressgateway-78c6d8b8d7-pdtb8 --tail=0 -f
[2019-02-19T20:38:07.655Z] "GET /HTTP/1.1" 404 NR 0 0 1 - "10.20.3.129" "curl/7.63.0" "154408c8-e941-4b07-b1ef-febe62fe8bd6" "bookinfo.example.com:31380" "-" - - 10.20.3.147:80 10.20.3.129:41574
+1
narun4sk on Feb 19, 2019

It’s a very basic use case, so I’m assuming there must be a workaround. Can I somehow configure istio to accept traffic on the NodePort?

It seems I cannot set VirtualService.spec.hosts like this:

  hosts:
  - "bookinfo.example.com:31380"

Istio returns validation error:

admission webhook “pilot.validation.istio.io” denied the request: configuration is invalid: domain name “example.com:31380” invalid (label"com:31380" invalid)

Currently I confirm that I have to explicitly set the Host header, to make it work…

$ curl -LsD- -o /dev/null bookinfo.example.com:31380
HTTP/1.1 404 Not Found
date: Tue, 19 Feb 2019 19:21:59 GMT
server: envoy
content-length: 0

$ curl -LsD- -o /dev/null -H 'Host: bookinfo.example.com' bookinfo.example.com:31380
HTTP/1.1 200 OK
content-type: text/html; charset=utf-8
content-length: 1836
server: envoy
date: Tue, 19 Feb 2019 19:21:48 GMT
x-envoy-upstream-service-time: 3
+1
narun4sk on Feb 19, 2019

By default, the host in header is bookinfo.example.com:31380

+1
hzxuzhonghu on Feb 19, 2019

As can be seen from above, the routable domains are "
domains": [ "bookinfo.example.com", "bookinfo.example.com:80" ],

so this is expected, you can access by specify -H"host: bookinfo.example.com"

+1
hzxuzhonghu on Feb 19, 2019
Read more comments on GitHub
← istio: 1.6 istio-proxy SDS not updating certificates SSLV3_ALERT_CERTIFICATE_EXPIRED
istio: 503 during VirtualService update in 1.8 →