istio: ingressgateway can't access https service.

Describe the bug I tried to set up a gateway and virtual service for an https service. But the gateway can only send clear http request to the https service. And the https service return 400 Bad Request - The plain HTTP request was sent to HTTPS port.

Expected behavior https request should be sent from gateway to https service.

Steps to reproduce the bug 1, Install the my-nginx project from istio sample. This is a https service. 2, add gateway as below

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: nginx-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "*"
    tls:
      mode: SIMPLE #enables HTTPS on this port
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

3, add virtual service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-nginx
spec:
  hosts:
  - "*"
  gateways:
  - nginx-gateway
  http:
  - match:  
    - uri:
        prefix: "/"  
    route:
    - destination:
        host: my-nginx
        port:
          number: 443

4, run test run below command. I use NodePort. The 10.0.0.27 is the node IP, 31390 is the node port.

curl https://10.0.0.27:31390 -k

run result

<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.11.3</center>
</body>
</html>

Version What version of istio and Kubernetes are you using? Use istioctl version and kubectl version istioctl version

Version: 1.0.0
GitRevision: 3a136c90ec5e308f236e0d7ebb5c4c5e405217f4
User: root@71a9470ea93c
Hub: gcr.io/istio-release
GolangVersion: go1.10.1
BuildStatus: Clean

kubectl version

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Is Istio Auth enabled or not? Did you install the stable istio.yaml, istio-auth.yaml… or if using the Helm chart please provide full command line input.

kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
helm install install/kubernetes/helm/istio --name istio --namespace istio-system --set tracing.enabled=true

Environment Which environment, cloud vendor, OS, etc are you using?

Linux k8s-m 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Cluster state If you’re running on Kubernetes, consider following the instructions to generate “istio-dump.tar.gz”, then attach it here by dragging and dropping the file onto this issue.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 40 (17 by maintainers)

Most upvoted comments

@kongyi-ibm Yes, terminating TLS and originating TLS by the gateway should work. You just create a DestinationRule to your service, and specify tls mode as SIMPLE. Something like:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-tls-for-my-service
spec:
  host: myservice
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE # initiates HTTPS

@vadimeisenbergibm Hi would you please add example to istio document for SSL termination at ingress and SSL originatin at gateway to https service. I tried many config but I could run just one scenario 1- GATEWAY SSL termination (SIMPLE) 2- Virtual service http and not tls 3- DR tls mode : SIMPLE 4- PA : Permessive.
I expect that PA STRICT must work but it does not work. I think add this scenario to https://istio.io/latest/docs/ops/common-problems/network-issues/#tls-configuration-mistakes will be useful

@lubinson @ijsnellf let me describe how to configure it, will submit a PR to istio.io.

Thank you for the propt reply @vadimeisenbergibm ! We found that the best practice to achieve this properly would be with mTLS, namely by referencing the certificate for the running service, although we’ve run into another issue which we opened via #24908

@kish3007 Sorry, I did not see your questions.

Regarding the TLS services - I checked and it looks like neither Grafana nor Kiali show TLS services. If so, it seems that in order to monitor a service and to collect HTTP metrics on it, you have to perform TLS termination by the ingress gateway, and then to perform TLS origination again on the route from the ingress gateway to the service. For that, you have to mount the service certificate/private key in the ingress gateway pod which is not ideal, or to use Secret Discovery Service.

Alternatively, you can change the HTTPS service to become an HTTP one, let the ingress gateway perform TLS termination, and use Istio mutual TLS to encrypt the traffic to the service inside the mesh.

@kish3007 Please note that in the TLS passthrough mode you cannot use http routes, you can only use tls, see https://istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/#configure-an-ingress-gateway.

Also note that the traffic is TLS, Istio does not see the encrypted HTTP traffic, so Istio does not show it as HTTP in the dashboard.

Except the SSL passthrough, do we have any other solution can let ingressgateway to terminal SSL and send https request to backend https service ?

I want to implement the solution like this ( it is most like the enable secret backend in ingress ).

https request --> ingressgateway --> https service


Gateway: spec: selector: istio: ingressgateway servers:

  • hosts:
    • ‘*’ port: name: https number: 443 protocol: HTTPS tls: mode: SIMPLE privateKey: /etc/istio/ingressgateway-certs/tls.key serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Virtualservice: spec: gateways:

  • mygateway hosts:
  • ‘*’ http:
  • match:
    • uri: prefix: /search route:
    • destination: host: search.default.svc.cluster.local port: number: 3738

Service definition: pec: clusterIP: 10.0.0.216 externalTrafficPolicy: Cluster ports:

  • name: https nodePort: 31702 port: 3738 protocol: TCP targetPort: 3738 selector: app: search sessionAffinity: None type: NodePort status: loadBalancer: {}

But when I use curl to access, the ingress always report 503. But I can curl from the envoy sidecar and get correct response

[root@pe103 ~]# curl -v -k https://9.112.245.103:31390/search/admin/resources/health/ping

  • About to connect() to 9.112.245.103 port 31390 (#0)
  • Trying 9.112.245.103…
  • Connected to 9.112.245.103 (9.112.245.103) port 31390 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • skipping SSL peer certificate verification
  • Server certificate:
  •   subject: CN=*
    
  •   start date: Jun 09 15:40:43 2017 GMT
    
  •   expire date: May 26 03:41:13 2027 GMT
    
  •   common name: *
    
  •   issuer: CN=testcert.watsoncommerce.ibm.com
    

GET /search/admin/resources/health/ping HTTP/1.1 User-Agent: curl/7.29.0 Host: 9.112.245.103:31390 Accept: /

< HTTP/1.1 503 Service Unavailable < content-length: 57 < content-type: text/plain < date: Mon, 26 Nov 2018 16:57:17 GMT < server: envoy < x-envoy-upstream-service-time: 8 <

  • Connection #0 to host 9.112.245.103 left intact

[root@pe103 ~]# istioctl authn tls-check search.default.svc.cluster.local HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE search.default.svc.cluster.local:3738 OK HTTP HTTP - -


from ingressgateway log, it looks like the ingressgateway can find target backend, but just can not generate connection

[2018-11-26T16:57:17.946Z] “GET /search/admin/resources/health/ping HTTP/1.1” 503 - 0 57 12 8 “10.1.1.0” “curl/7.29.0” “8c92b97e-933a-9590-b1d8-5584b0be636c” “9.112.245.103:31390” “10.1.146.230:3738”

@lubinson Here is the example I wrote https://preliminary.istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/

Note that you have to use PASSTHROUGH tls mode since the gateway must not perform TLS termination. Also note that the VirtualService should have tls match by sni_hosts.