istio: ingressgateway can't access https service with TLS termination.

Bug description I tried to set up a gateway and virtual service for an https service. used SIMPLE mode for the TLS termination in gateway. And the https service return 404 Not Found. But if i use PASSTHROUGH then https service return proper response.

Expected behavior https request should be sent from gateway to https service with SIMPLE mode.

Steps to reproduce the bug 1, Install the my-nginx project from istio sample. This is a https service. https://istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/#configure-an-ingress-gateway

2, add gateway as below

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "simplessl"
    hosts:
    - nginx.example.com

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - nginx.example.com
  gateways:
  - mygateway
  tls:
  - match:
    - port: 443
      sni_hosts:
      - nginx.example.com
    route:
    - destination:
        host: my-nginx
        port:
          number: 443
  1. run curl command
curl https://nginx.example.com -k -v

run result

* Rebuilt URL to: https://nginx.example.com/
*   Trying 104.42.42.106...
* Connected to nginx.example.com (104.42.42.106) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 594 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: nginx.example.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=Denial,L=Springfield,O=Dis,CN=nginx.example.com
*        start date: Wed, 24 Apr 2019 10:08:21 GMT
*        expire date: Sun, 03 May 2020 10:08:21 GMT
*        issuer: C=US,ST=Denial,O=Dis,CN=nginx.example.com
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: nginx.example.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< location: https://nginx.example.com/
< date: Wed, 24 Apr 2019 10:22:18 GMT
< server: istio-envoy
< content-length: 0

Version (include the output of istioctl version --remote and kubectl version) istioctl version

version.BuildInfo{Version:"1.1.2", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"35adf5bb-5570-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.1"}

kubectl version

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"51dd616cdd25d6ee22c83a858773b607328a18ec", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?

for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
kubectl apply -f install/kubernetes/istio-demo.yaml

Configure a TLS ingress gateway using SDS

$ helm template install/kubernetes/helm/istio/ --name istio \
--namespace istio-system -x charts/gateways/templates/deployment.yaml \
--set gateways.istio-egressgateway.enabled=false \
--set gateways.istio-ingressgateway.sds.enabled=true > \
$HOME/istio-ingressgateway.yaml
$ kubectl apply -f $HOME/istio-ingressgateway.yaml

Environment where bug was observed (cloud vendor, OS, etc)

Linux aks-agentpool-14689735-0 4.15.0-1037-azure #39~16.04.1-Ubuntu SMP Tue Jan 15 17:20:47 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 38 (17 by maintainers)

Most upvoted comments

@yonwon01 I do now work on Istio starting from 2020, sorry. The issue was closed at some point.

@vadimeisenbergibm Thank you very much for your time and patience.

That’s curious. I tested changing (1) The prefix indent and (2) Removing the istio: ingressgateway. One of those things made it work; too bad I did it at the same time.

Reverting back both changes is not causing the 404 anymore (Is there some caching mechanism with the rules in the Istio proxy?)

Two more things to try:

  1. Indent prefix to the right, under uri
  2. Set the hosts in the gateway to istio.iterativo.do

I’m having the same problem; I’m getting 404 when using HTTPS. Port 80 working great.

Config files

apiVersion: v1
kind: Service
metadata:
  name: odoo-service
  labels:
    app: odoo
    istio: ingressgateway
  namespace: default
spec:
  selector:
    app: odoo
  ports:
  - name: https-odoo
    port: 8069
    targetPort: 8069
  - name: http-odoo
    port: 8069
    targetPort: 8069
  - name: http-longpolling
    port: 8072
    targetPort: 8072
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istio-odoo
spec:
  gateways:
  - istio-odoo.default
  hosts:
  - istio.iterativo.do
  http:
  - match:
    - uri:
      prefix: /
    route:
    - destination:
        host: odoo-service.default.svc.cluster.local
        port:
          number: 8069
  - match:
    - uri:
      prefix: /longpolling
    route:
    - destination:
        host: odoo-service.default.svc.cluster.local # Destination service, must match service name
        port:
          number: 8072
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-odoo # Individual name (we'll use this later in VirtualServer)
  namespace: default # Namespace
spec:
  selector:
    istio: ingressgateway # Specify the ingressgateway created for us by GKE
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP2
  - hosts:
    - '*'
    port:
      name: https-default
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Curl result

* Rebuilt URL to: https://istio.iterativo.do/
*   Trying 34.74.1.138...
* TCP_NODELAY set
* Connected to istio.iterativo.do (34.74.1.138) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=iterativo.do
*  start date: Aug 19 03:06:20 2019 GMT
*  expire date: Nov 17 03:06:20 2019 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ea197b7e80)
> GET / HTTP/1.1
> Host: istio.iterativo.do
> User-Agent: curl/7.52.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
< location: https://istio.iterativo.do/
< date: Mon, 19 Aug 2019 05:43:39 GMT
< server: istio-envoy
<
* Curl_http_done: called premature == 0
* Connection #0 to host istio.iterativo.do left intact

Any updates on what I might be doing wrong?