istio: TLS termination of HTTPS on ingress gateway appears to be broken
Bug description
Tutorial https://istio.io/latest/docs/tasks/traffic-management/ingress/secure-ingress/ is about deploying httpbin to Kubernetes, and configuring Istio’s ingress gateway for HTTPS. I followed exactly that tutorial and the curl client fails to connect.
# create a namespace for the httpbin application
kubectl create ns httpbin
kubectl label namespace httpbin istio-injection=enabled
kubectl label namespace httpbin app=httpbin
# create a secret in istio-system namespace
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
openssl req -out httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"
openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in httpbin.example.com.csr -out httpbin.example.com.crt
kubectl create -n istio-system secret tls httpbin-tls-certificate --key=httpbin.example.com.key --cert=httpbin.example.com.crt
# deploy httpbin
kubectl apply -f httpbin.yaml
and httpbin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: httpbin
namespace: httpbin
labels:
app: httpbin
---
apiVersion: v1
kind: Service
metadata:
name: httpbin
namespace: httpbin
labels:
app: httpbin
spec:
ports:
- name: http
port: 80
targetPort: 80
selector:
app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: httpbin
namespace: httpbin
spec:
replicas: 1
selector:
matchLabels:
app: httpbin
version: v1
template:
metadata:
labels:
app: httpbin
version: v1
spec:
serviceAccountName: httpbin
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin
ports:
- containerPort: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: httpbin
namespace: httpbin
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "httpbin.example.com"
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: httpbin-tls-certificate
hosts:
- "httpbin.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
namespace: httpbin
spec:
hosts:
- "httpbin.example.com"
gateways:
- httpbin
http:
- match:
- uri:
prefix: /status
- uri:
prefix: /delay
route:
- destination:
host: httpbin
port:
number: 80
INGRESS_HOST = kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
INGRESS_PORT = kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}'
INGRESS_PORT_SECURE = kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].port}'
Plaintext works: curl -i -HHost:httpbin.example.com "http://$(INGRESS_HOST):$(INGRESS_PORT)/status/200"
returns status OK.
HTTPS does not work curl -v -i -HHost:httpbin.example.com --cacert example.com.crt "https://$(INGRESS_HOST):$(INGRESS_PORT_SECURE)/status/200
returns
* Trying [redacted IP]...
* Connected to [redacted domain].us-west-2.elb.amazonaws.com ([redacted IP]) port 443 (#0)
* found 1 certificates in example.com.crt
* found 526 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* gnutls_handshake() failed: The TLS connection was non-properly terminated.
* Closing connection 0
[ ] Docs [ ] Installation [X] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [ ] User Experience [ ] Developer Infrastructure
Expected behavior
The tutorial works
Steps to reproduce the bug
$ istioctl version --remote
client version: 1.7.0
control plane version: 1.7.0
data plane version: 1.7.0 (3 proxies)
$ kubectl version --short
Client Version: v1.17.7-eks-bffbac
Server Version: v1.17.9-eks-4c6976
How was Istio installed?
istioctl install --set profile=demo
Environment where bug was observed (cloud vendor, OS, etc)
- EKS 1.17
- Istio ingress gateway logs:
Version 1.7.0-2022348138e47498c4b54995b4cb5a1656817c4e-Clean
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 3
- Comments: 16 (8 by maintainers)
https://istio.io/latest/faq/security/#verify-mtls-encryption describes this, but we should probably improve it… I will follow up with some folks. Thanks!