istio: TLS termination of HTTPS on ingress gateway appears to be broken

Bug description

Tutorial https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/ is about deploying httpbin to Kubernetes, and configuring Istio’s ingress gateway for HTTPS. I followed exactly that tutorial and the curl client fails to connect.

# create a namespace for the httpbin application
kubectl create ns httpbin
kubectl label namespace httpbin istio-injection=enabled
kubectl label namespace httpbin app=httpbin

# create a secret in istio-system namespace
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
openssl req -out httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in httpbin.example.com.csr -out httpbin.example.com.crt

kubectl create -n istio-system secret tls httpbin-tls-certificate --key=httpbin.example.com.key --cert=httpbin.example.com.crt

# deploy httpbin
kubectl apply -f httpbin.yaml

and httpbin.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    app: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 80
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin
  namespace: httpbin
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "httpbin.example.com"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: httpbin-tls-certificate
    hosts:
      - "httpbin.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: httpbin
spec:
  hosts:
  - "httpbin.example.com"
  gateways:
  - httpbin
  http:
  - match:
    - uri:
        prefix: /status
    - uri:
        prefix: /delay
    route:
    - destination:
        host: httpbin
        port:
          number: 80
INGRESS_HOST = kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
INGRESS_PORT =  kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}'
INGRESS_PORT_SECURE = kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}'

Plaintext works: curl -i -HHost:httpbin.example.com "http://$(INGRESS_HOST):$(INGRESS_PORT)/status/200" returns status OK.

HTTPS does not work curl -v -i -HHost:httpbin.example.com --cacert example.com.crt "https://$(INGRESS_HOST):$(INGRESS_PORT_SECURE)/status/200 returns

*   Trying [redacted IP]...
* Connected to [redacted domain].us-west-2.elb.amazonaws.com ([redacted IP]) port 443 (#0)
* found 1 certificates in example.com.crt
* found 526 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: The TLS connection was non-properly terminated.
* Closing connection 0

[ ] Docs [ ] Installation [X] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure

Expected behavior

The tutorial works

Steps to reproduce the bug

$ istioctl version --remote
client version: 1.7.0
control plane version: 1.7.0
data plane version: 1.7.0 (3 proxies)

$ kubectl version --short
Client Version: v1.17.7-eks-bffbac
Server Version: v1.17.9-eks-4c6976

How was Istio installed?

istioctl install --set profile=demo

Environment where bug was observed (cloud vendor, OS, etc)

  • EKS 1.17
  • Istio ingress gateway logs: Version 1.7.0-2022348138e47498c4b54995b4cb5a1656817c4e-Clean

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 3
  • Comments: 16 (8 by maintainers)

Most upvoted comments

https://istio.io/latest/faq/security/#verify-mtls-encryption describes this, but we should probably improve it… I will follow up with some folks. Thanks!