istio: Unable to configure DestinationRules for routes from Gateway with 1.1.0-rc.0

Describe the bug I created a service without the istio-proxy sidecar and wanted to make this available using the istio-ingressgateway. To allows calls to this service I have to configure a DestinationRule that disables TLS for this connection as the service will not understand mTLS. I configured the DestinationRule to use a dedicated subset name plain.

While the route is properly pushed to the ingress gateway pointing to outbound|8080|plain|testserver.foo.svc.cluster.local, the cluster doesn’t appear there.

Additionally there is another cluster now that doesn’t follow the naming scheme: outbound_.8080_._.testserver.foo.svc.cluster.local

I tried the same configuration with Istio 1.0.5 and that worked.

I can’t get away from the feeling that I have done something wrong, but the fact that this config works with 1.0 and I can see no error made me create this Issue.

Expected behavior It should be possible to expose services that don’t have an istio-proxy via the istio-ingressgateway.

Steps to reproduce the bug Install Istio 1.1.0-rc.0 and apply this configuration:

apiVersion: v1
kind: Namespace
metadata:
  name: foo
  labels:
    istio-injection: disabled

---
apiVersion: v1
kind: Namespace
metadata:
  name: foo2
  labels:
    istio-injection: enabled

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway-http
  namespace: foo
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - test.server

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: testserver
  namespace: foo
spec:
  gateways:
    - gateway-http
    - mesh
  hosts:
    - test.server
  http:
    - match:
        - uri:
            prefix: "/get"
      route:
        - destination:
            host: testserver.foo.svc.cluster.local
            subset: plain
            port:
              number: 8080
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: testserver
  namespace: foo
spec:
  host: testserver.foo.svc.cluster.local
  subsets:
    - name: plain
      trafficPolicy:
        connectionPool:
          tcp:
            connectTimeout: 2.000s
        tls: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testserver
  namespace: foo
spec:
  selector:
    matchLabels:
      app: testserver
  replicas: 2
  template:
    metadata:
      labels:
        app: testserver
    spec:
      containers:
        - name: testserver
          image: kennethreitz/httpbin
          ports:
            - containerPort: 80
              name: http-testserver
---
apiVersion: v1
kind: Service
metadata:
  name: testserver
  namespace: foo
spec:
  selector:
    app: testserver
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: http-testserver


---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testcurl
  namespace: foo2
spec:
  selector:
    matchLabels:
      app: testcurl
  replicas: 1
  template:
    metadata:
      labels:
        app: testcurl
    spec:
      containers:
        - name: testcurl
          image: tutum/curl
          command:
            - sleep
            - "100000"

The service to expose is httpbin and resides in the namespace foo available as testserver.foo.svc.cluster.local. The service exposes the application at port 8080. A VirtualService for the host name test.server should route to this service.

After applying this configuration check the config of the ingress gateway. The route properly routes to the correct cluster:

   "dynamic_route_configs": [
    {
     "version_info": "2019-02-26T16:22:48Z/4",
     "route_config": {
      "name": "http.80",
      "virtual_hosts": [
       {
        "name": "test.server:80",
        "domains": [
         "test.server",
         "test.server:80"
        ],
        "routes": [
         {
          "match": {
           "prefix": "/get"
          },
          "route": {
           "cluster": "outbound|8080|plain|testserver.foo.svc.cluster.local",

However, this cluster is never pushed to the ingress gateway:

curl -s localhost:15000/clusters|cut -d ':' -f 1|uniq|sort|grep foo
outbound_.8080_._.testserver.foo.svc.cluster.local
outbound|8080||testserver.foo.svc.cluster.local

Hence I cannot call this service via the Ingress Gateway and get a 503 (which means it hits the VirtualService because it doesn’t get a 404):

curl -v localhost/get -H 'host:test.server'
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /get HTTP/1.1
> host:test.server
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< date: Tue, 26 Feb 2019 16:35:09 GMT
< server: istio-envoy
< content-length: 0
<
* Connection #0 to host localhost left intact

The configuration above has a consumer deployment that has a sidecar. As the VirtualService is also targeted to the mesh gateway, I can see there that the cluster is properly pushed to this sidecar:

curl -s localhost:15000/clusters|cut -d ':' -f 1|uniq|sort|grep foo
outbound|8080|plain|testserver.foo.svc.cluster.local
outbound|8080||testserver.foo.svc.cluster.local

Here I also don’t see that strange looking cluster name with the _.. Additionally I can call the VirtualService successfully from the consumer deployment:

root@testcurl-9ff7dd55d-w5z9t:/# curl -v 1.2.3.4/get -H 'host: test.server'
* Hostname was NOT found in DNS cache
*   Trying 1.2.3.4...
* Connected to 1.2.3.4 (1.2.3.4) port 80 (#0)
> GET /get HTTP/1.1
> User-Agent: curl/7.35.0
> Accept: */*
> host: test.server
>
< HTTP/1.1 200 OK
* Server envoy is not blacklisted
< server: envoy
< date: Tue, 26 Feb 2019 16:32:28 GMT
< content-type: application/json
< content-length: 807
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 16
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "test.server",
    "User-Agent": "curl/7.35.0",
    "X-B3-Sampled": "1",
    "X-B3-Spanid": "211df12a8f8acac0",
    "X-B3-Traceid": "a8b1a1928c62b1e3211df12a8f8acac0",
    "X-Envoy-Decorator-Operation": "testserver.foo.svc.cluster.local:8080/get*",
    "X-Istio-Attributes": "CigKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIMEgp0ZXN0c2VydmVyCiYKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEgUSA2Zvbwo6Cgpzb3VyY2UudWlkEiwSKmt1YmVybmV0ZXM6Ly90ZXN0Y3VybC05ZmY3ZGQ1NWQtdzV6OXQuZm9vMgo8ChdkZXN0aW5hdGlvbi5zZXJ2aWNlLnVpZBIhEh9pc3RpbzovL2Zvby9zZXJ2aWNlcy90ZXN0c2VydmVyCj4KGGRlc3RpbmF0aW9uLnNlcnZpY2UuaG9zdBIiEiB0ZXN0c2VydmVyLmZvby5zdmMuY2x1c3Rlci5sb2NhbA=="
  },
  "origin": "10.1.4.207",
  "url": "http://test.server/get"
}

Version

k version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Istio 1.1.0-rc.0 The Ingress Gateway identifies like this:

"version": "fc273e117a6ed875c7cb9b3c3251c8d982bc7196/1.10.0-dev/Clean/RELEASE/BoringSSL"

The sidecar of the consumer deployment shows the same version

Installation helm install ~/istio/istio-1.1.0-rc.0/install/kubernetes/helm/istio-init --name istio-init --namespace istio-system -f ~/istio/istio-1.1.0-rc.0/install/kubernetes/helm/istio/values-istio-auth-mcp.yaml --set global.mtls.enabled=true --set global.tracing.enabled=true --set pilot.traceSampling=100.0 --set tracing.enabled=true --set grafana.enabled=true --set global.useMCP=true

helm install ~/istio/istio-1.1.0-rc.0/install/kubernetes/helm/istio --name istio --namespace istio-system -f ~/istio/istio-1.1.0-rc.0/install/kubernetes/helm/istio/values-istio-auth-mcp.yaml --set global.mtls.enabled=true --set global.tracing.enabled=true --set pilot.traceSampling=100.0 --set tracing.enabled=true --set grafana.enabled=true --set global.useMCP=true

Environment Running locally on Docker for Mac.

Cluster state The attached file contains the configuration of the Ingress Gateway on 1.1 and 1.0. Additionally it contains the logs of the Pilot on 1.1.0-rc.0. logs.zip