istio: ingressgateway can't access https service.
Describe the bug I tried to set up a gateway and virtual service for an https service. But the gateway can only send clear http request to the https service. And the https service return 400 Bad Request - The plain HTTP request was sent to HTTPS port.
Expected behavior https request should be sent from gateway to https service.
Steps to reproduce the bug 1, Install the my-nginx project from istio sample. This is a https service. 2, add gateway as below
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: nginx-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "*"
tls:
mode: SIMPLE #enables HTTPS on this port
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
3, add virtual service
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-nginx
spec:
hosts:
- "*"
gateways:
- nginx-gateway
http:
- match:
- uri:
prefix: "/"
route:
- destination:
host: my-nginx
port:
number: 443
4, run test run below command. I use NodePort. The 10.0.0.27 is the node IP, 31390 is the node port.
curl https://10.0.0.27:31390 -k
run result
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.11.3</center>
</body>
</html>
Version
What version of istio and Kubernetes are you using? Use istioctl version
and kubectl version
istioctl version
Version: 1.0.0
GitRevision: 3a136c90ec5e308f236e0d7ebb5c4c5e405217f4
User: root@71a9470ea93c
Hub: gcr.io/istio-release
GolangVersion: go1.10.1
BuildStatus: Clean
kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"5ca598b4ba5abb89bb773071ce452e33fb66339d", GitTreeState:"clean", BuildDate:"2018-06-06T08:00:59Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Is Istio Auth enabled or not? Did you install the stable istio.yaml, istio-auth.yaml… or if using the Helm chart please provide full command line input.
kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
helm install install/kubernetes/helm/istio --name istio --namespace istio-system --set tracing.enabled=true
Environment Which environment, cloud vendor, OS, etc are you using?
Linux k8s-m 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Cluster state If you’re running on Kubernetes, consider following the instructions to generate “istio-dump.tar.gz”, then attach it here by dragging and dropping the file onto this issue.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 40 (17 by maintainers)
@kongyi-ibm Yes, terminating TLS and originating TLS by the gateway should work. You just create a DestinationRule to your service, and specify
tls
mode as SIMPLE. Something like:@vadimeisenbergibm Hi would you please add example to istio document for SSL termination at ingress and SSL originatin at gateway to https service. I tried many config but I could run just one scenario 1- GATEWAY SSL termination (SIMPLE) 2- Virtual service http and not tls 3- DR tls mode : SIMPLE 4- PA : Permessive.
I expect that PA STRICT must work but it does not work. I think add this scenario to https://istio.io/latest/docs/ops/common-problems/network-issues/#tls-configuration-mistakes will be useful
@lubinson @ijsnellf let me describe how to configure it, will submit a PR to istio.io.
Thank you for the propt reply @vadimeisenbergibm ! We found that the best practice to achieve this properly would be with mTLS, namely by referencing the certificate for the running service, although we’ve run into another issue which we opened via #24908
@kish3007 Sorry, I did not see your questions.
Regarding the TLS services - I checked and it looks like neither Grafana nor Kiali show TLS services. If so, it seems that in order to monitor a service and to collect HTTP metrics on it, you have to perform TLS termination by the ingress gateway, and then to perform TLS origination again on the route from the ingress gateway to the service. For that, you have to mount the service certificate/private key in the ingress gateway pod which is not ideal, or to use Secret Discovery Service.
Alternatively, you can change the HTTPS service to become an HTTP one, let the ingress gateway perform TLS termination, and use Istio mutual TLS to encrypt the traffic to the service inside the mesh.
@kish3007 Please note that in the TLS passthrough mode you cannot use
http
routes, you can only usetls
, see https://istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/#configure-an-ingress-gateway.Also note that the traffic is TLS, Istio does not see the encrypted HTTP traffic, so Istio does not show it as HTTP in the dashboard.
Except the SSL passthrough, do we have any other solution can let ingressgateway to terminal SSL and send https request to backend https service ?
I want to implement the solution like this ( it is most like the enable secret backend in ingress ).
https request --> ingressgateway --> https service
Gateway: spec: selector: istio: ingressgateway servers:
Virtualservice: spec: gateways:
Service definition: pec: clusterIP: 10.0.0.216 externalTrafficPolicy: Cluster ports:
But when I use curl to access, the ingress always report 503. But I can curl from the envoy sidecar and get correct response
[root@pe103 ~]# curl -v -k https://9.112.245.103:31390/search/admin/resources/health/ping
< HTTP/1.1 503 Service Unavailable < content-length: 57 < content-type: text/plain < date: Mon, 26 Nov 2018 16:57:17 GMT < server: envoy < x-envoy-upstream-service-time: 8 <
[root@pe103 ~]# istioctl authn tls-check search.default.svc.cluster.local HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE search.default.svc.cluster.local:3738 OK HTTP HTTP - -
from ingressgateway log, it looks like the ingressgateway can find target backend, but just can not generate connection
[2018-11-26T16:57:17.946Z] “GET /search/admin/resources/health/ping HTTP/1.1” 503 - 0 57 12 8 “10.1.1.0” “curl/7.29.0” “8c92b97e-933a-9590-b1d8-5584b0be636c” “9.112.245.103:31390” “10.1.146.230:3738”
@lubinson Here is the example I wrote https://preliminary.istio.io/docs/examples/advanced-gateways/ingress-sni-passthrough/
Note that you have to use
PASSTHROUGH
tls
mode
since the gateway must not perform TLS termination. Also note that theVirtualService
should havetls
match bysni_hosts
.