istio: GateWay Ssl Error

Got exact the same issue:

SSL Certificates are in place: HTTP flow works fine

But! once I provision both Gateways - HTTP and HTTPS. SSL connection begin to work removing HTTP leads to same error (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to mydomain.com

@xlanor I solved it by creating the certificate manually and then uploading it as a secret. It seems the issue is in the certIssuer. I tested this in AWS and GCP. Generate your own certificates as secrets and reference them in the gateway. Remember to put it in the same namespace as the sitio Gateway (sitio-system).

I still experience this issue as well (istio version 1.0.0)… (I had performed basic troubleshooting above in this thread)

ingressgateway service

istio-ingressgateway       LoadBalancer   10.100.224.41    <REDACTED>...   80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:31337/TCP,8060:30341/TCP,15030:30834/TCP,15031:30600/TCP   19h

ingressgateway pod details:

kubectl describe pods/istio-ingressgateway-67995c486c-w2pcc -n istio-system
Name:           istio-ingressgateway-67995c486c-w2pcc
Namespace:      istio-system
Node:           ip-172-27-22-150.ec2.internal/172.27.22.150

(OK) Attempt 1: try to connect from a worker to the ingressgateway ClusterIp:

curl -k -vvvv --resolve httpbin.example.com:443:10.100.224.41  https://httpbin.example.com/status/200
<....>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: envoy
< date: Mon, 13 Aug 2018 08:23:10 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: *
< access-control-allow-credentials: true
< content-length: 0
< x-envoy-upstream-service-time: 5

(OK)Attempt 2: trying to connect directly to the host port (over https):

curl -k -vvvv --header 'Host: httpbin.example.com' --resolve httpbin.example.com:31390:172.27.12.33  https://httpbin.example.com:31390/status/200

* Added httpbin.example.com:31390:172.27.12.33 to DNS cache
* Hostname httpbin.example.com was found in DNS cache
*   Trying 172.27.12.33...
* TCP_NODELAY set
* Connected to httpbin.example.com (172.27.12.33) port 31390 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Denial; L=Springfield; O=Dis; CN=*.example.com
*  start date: Aug 10 12:16:35 2018 GMT
*  expire date: Aug 20 12:16:35 2019 GMT
*  issuer: C=US; ST=Denial; O=Dis; CN=*.example.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5631c572b8e0)
> GET /status/200 HTTP/2
> Host: httpbin.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< server: envoy
< date: Mon, 13 Aug 2018 08:45:41 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: *
< access-control-allow-credentials: true
< content-length: 0
< x-envoy-upstream-service-time: 5
<
* Connection #0 to host httpbin.example.com left intact

(OK) Attempt 3: connecting via the ELB (over HTTP):

curl -v http://httpbin.example.com/status/200
*   Trying <REDACTED>...
* TCP_NODELAY set
* Connected to httpbin.example.com (<REDACTED>) port 80 (#0)
> GET /status/200 HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< server: envoy
< date: Mon, 13 Aug 2018 08:52:07 GMT
< content-type: text/html; charset=utf-8
< access-control-allow-origin: *
< access-control-allow-credentials: true
< content-length: 0
< x-envoy-upstream-service-time: 5
<
* Connection #0 to host httpbin.example.com left intact

(NOT OK) Attempt 4: connecting via the ELB (over HTTPS):

$ curl -k  -vvvv  https://httpbin.example.com/status/200
*   Trying <REDACTED>...
* TCP_NODELAY set
* Connected to httpbin.example.com (<REDACTED>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.example.com
*  start date: Dec 22 00:00:00 2017 GMT
*  expire date: Jan 22 12:00:00 2019 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET /status/200 HTTP/1.1
> Host: httpbin.example.com
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 408 REQUEST_TIMEOUT
< Content-Length:0
< Connection: Close
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

and finally my ELB configuration, created automatically by kubernetes LoadBalancer Service:

{
    "LoadBalancerDescriptions": [
        {
            "LoadBalancerName": "<REDACTED>",
            "DNSName": "<REDACTED>.us-east-1.elb.amazonaws.com",
            "CanonicalHostedZoneName": "<REDACTED>.us-east-1.elb.amazonaws.com",
            "CanonicalHostedZoneNameID": "Z35SXDOTRQ7X7K",
            "ListenerDescriptions": [
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 80,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31380
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "HTTPS",
                        "LoadBalancerPort": 443,
                        "InstanceProtocol": "HTTPS",
                        "InstancePort": 31390,
                        "SSLCertificateId": "arn:aws:acm:us-east-1:709693308983:certificate/<REDACTED>"
                    },
                    "PolicyNames": [
                        "ELBSecurityPolicy-2016-08"
                    ]
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 31400,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31400
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 15011,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 31337
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 8060,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30341
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 15030,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30834
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "Protocol": "TCP",
                        "LoadBalancerPort": 15031,
                        "InstanceProtocol": "TCP",
                        "InstancePort": 30600
                    },
                    "PolicyNames": []
                }
            ],
            "Policies": {
                "AppCookieStickinessPolicies": [],
                "LBCookieStickinessPolicies": [],
                "OtherPolicies": [
                    "ELBSecurityPolicy-2016-08"
                ]
            },
            "BackendServerDescriptions": [],
            "AvailabilityZones": [
                "us-east-1a",
                "us-east-1c",
                "us-east-1d"
            ],
            "Subnets": [
                "subnet-975ffdcb",
                "subnet-9a12beb4",
                "subnet-b696e7fc"
            ],
            "VPCId": "vpc-9e0ef9e4",
            "Instances": [
                {
                    "InstanceId": "i-021a786c4362a0d6a"
                },
                {
                    "InstanceId": "i-02b69c0d561dc27bc"
                },
                {
                    "InstanceId": "i-02caf5296ecedad61"
                },
                {
                    "InstanceId": "i-0681d728a799591a6"
                },
                {
                    "InstanceId": "i-0b2102f062df52bc0"
                },
                {
                    "InstanceId": "i-0e0fa4a4d802135cd"
                }
            ],
            "HealthCheck": {
                "Target": "TCP:31390",
                "Interval": 10,
                "Timeout": 5,
                "UnhealthyThreshold": 6,
                "HealthyThreshold": 2
            },
            "SourceSecurityGroup": {
                "OwnerAlias": "709693308983",
                "GroupName": "k8s-elb-a2abbdb149e2a11e8a6950acd49a3f59"
            },
            "SecurityGroups": [
                "sg-06cbef25aae63b86e"
            ],
            "CreatedTime": "2018-08-12T12:21:00.750Z",
            "Scheme": "internet-facing"
        }
    ]
}

This is my gateway configuration:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.example.com"
  - port:
      number: 443
      name: http
      protocol: HTTP
    hosts:
    - "*.example.com"
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "*.example.com"
  gateways:
  - httpbin
  http:
  - match:
    - port: 80
    route:
      - destination:
          host: httpbin.default.svc.cluster.local
          port:
            number: 8000
  - match:
    - port: 443
    route:
      - destination:
          host: httpbin.default.svc.cluster.local
          port:
            number: 8000

Other facts:

• All the ports are open from the ELB security group to the kubernetes workers
• HTTPS works at the ELB level without termination ie. if I use 31380 over tcp as a backend
• All HTTP connections to the application work from the remote

Any help is greatly appreciated. Please let me know if you need additional information!

UPDATE

it seems a problem with sni? The following configuration works all fine

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*.example.com"
  - port:
      number: 443
      name: http
      protocol: HTTP
    hosts:
    - "*"                          // NOTICE only * instead of *.example.com
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

Still same issue on 1.1.0 and same workarounds working describe by @edussx. Tested with File Mount-Based approach and using Secret Discovery Service.

Same problem here on 1.0.5. I’ve tried many different configurations and the only one that works is when both http and https are enabled on the gateway but I don’t want anything to be accessible on http. httpRedirect won’t suffice as that then causes a “too many redirects” issue when enabling the cloudflare proxy and we absolutely need cloudflare.

@frankbu @jsenon I am having the same issue.

K8s version: 1.9
ISTIO Version: 1.0.1

K8s cluster have E load balancer enabled. 

I followed this tutorial https://istio.io/docs/tasks/traffic-management/secure-ingress/ and my curl was not working if I only have port 443. It was giving me error: SSL_ERROR_SYSCALL

curl -v -HHost:ms-1234.byom-i344382.eu-central-1.x.x.com https://ms-1234.byom-i344382.eu-central-1.x.x.com:443/api/v2/predict --cacert /home/vagrant/exmaple/mtls-go-example/2_intermediate/certs/ca-chain.cert.pem

then I added port 80 and redirected the traffic to https and it started working.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ms-is-gateway
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
    hosts:
    - "*.byom-i344382.eu-central-1.x.x.com"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*.byom-i344382.eu-central-1.x.x.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ms-is
spec:
  hosts:
  - "*.byom-i344382.eu-central-1.x.x.com"
  gateways:
  - ms-is-gateway
  http:
  - match:
    - uri:
        exact: /api/v2/predict
    route:
    - destination:
        host: mlf-is.default.svc.cluster.local
        port:
          number: 53547
  - match:
    - headers:
        content-type:
          exact: application/grpc
    route:
    - destination:
        host: mlf-is.default.svc.cluster.local
        port:
          number: 9000

I’m also experiencing a similar issue.

In my case using an NLB in combination with HTTPS (I’ve tried an ELB as well with no better luck).

What I see is:

curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 18.210.31.22:443

and a very similar error is also thrown if I try to connect internally to the ClusterIp of the service:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 10.100.243.178:443

Certificates have been created as suggested using https://github.com/nicholasjackson/mtls-go-example

This is the result of the suggested troubleshooting steps:

1. Host would be an NLB address, port 443

$ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-certs

total 8
drwxrwxrwt 3 root root   80 Jul 19 08:41 .
drwxr-xr-x 1 root root 4096 Jul 19 08:41 ..
drwxr-xr-x 2 root root   40 Jul 19 08:41 ..2018_07_19_08_41_16.761205109
lrwxrwxrwx 1 root root   31 Jul 19 08:41 ..data -> ..2018_07_19_08_41_16.761205109

1. Logs from the ingress gateway show:

[2018-07-19 08:59:53.102][52][warning][config] external/envoy/source/server/listener_manager_impl.cc:254] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)
[2018-07-19 08:59:53.103][52][info][upstream] external/envoy/source/server/lds_api.cc:62] lds: add/update listener '0.0.0.0_443'
[2018-07-19 08:59:58.255][52][warning][config] external/envoy/source/server/listener_manager_impl.cc:254] adding listener '0.0.0.0:443': filter chain match rules require TLS Inspector listener filter, but it isn't configured, trying to inject it (this might fail if Envoy is compiled without it)
[2018-07-19 08:59:58.256][52][info][upstream] external/envoy/source/server/lds_api.cc:62] lds: add/update listener '0.0.0.0_443'

Thanks!

@anthonyhaussman I am glad it worked for you 😃

I still have the same issue in version 1.2.2. When I try to set a specific host in my Gateway, I have an SSL_ERROR_SYSCALL but it works great with a wildcard. I have a simple application who returns a token on a POST. Here are my tests. Set up working with hosts wildcard:

### With wildcard host
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: test
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https-test
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "wildcard-test-cmp-gg" ## Wildcard certificate *.test.cmp.gg
    hosts:
    - "*"
---


[~]> curl --resolve 'graphql-istio.test.cmp.gg:31390:kube-01.dev.cmp.com'  -vvv --header 'Host: graphql-istio.test.cmp.gg' -kX POST -d 'grant_type=password&username=test@cmp.com&password=ThePassord&client_id=1234&client_secret=1234' https://kube-01.dev.cmp.com:31390/oauth/token
Note: Unnecessary use of -X or --request, POST is already inferred.
* Resolve address 'kube-01.dev.cmp.com' found illegal!
* Couldn't parse CURLOPT_RESOLVE entry 'graphql-istio.test.cmp.gg:31390:kube-01.dev.cmp.com'!
*   Trying 192.168.1.1:31390...
* TCP_NODELAY set
* Connected to kube-01.dev.cmp.com (192.168.1.1) port 31390 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.test.cmp.gg
*  start date: May  2 21:16:09 2019 GMT
*  expire date: Jul 31 21:16:09 2019 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f2123fbf30)
> POST /oauth/token HTTP/2
> Host: graphql-istio.test.cmp.gg
> User-Agent: curl/7.65.1
> Accept: */*
> content-type: application/x-www-form-urlencoded
> Content-Length: 221
> 
* We are completely uploaded and fine
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 200 
< server: istio-envoy
< date: Fri, 05 Jul 2019 08:36:46 GMT
< content-type: application/json; charset=utf-8
< content-length: 1200
< cache-control: no-store
< pragma: no-cache
< x-envoy-upstream-service-time: 230
< backend-response-time: 227
< name: oauth
< version: 1
< endpoint: /oauth/token
< x-datadog-sampling-priority: -1
< 
{"access_token": OK}%

Set up not working with specific hosts:

### With specific host
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: test
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 443
      name: https-test
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: "wildcard-test-cmp-gg" ## Wildcard certificate *.test.cmp.gg
    hosts:
    - "graphql-istio.test.cmp.gg"
---

[~]> curl --resolve 'graphql-istio.test.cmp.gg:31390:192.168.1.1' -vvv --header 'Host: graphql-istio.test.cmp.gg' -kX POST -d 'grant_type=password&username=test@cmp.com&password=ThePassord&client_id=1234&client_secret=1234' https://kube-01.dev.cmp.com:31390/oauth/token
Note: Unnecessary use of -X or --request, POST is already inferred.
* Added graphql-istio.test.cmp.gg:31390:192.168.1.1 to DNS cache
*   Trying 192.168.1.1:31390...
* TCP_NODELAY set
* Connected to kube-01.dev.cmp.com (192.168.1.1) port 31390 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to kube-01.dev.cmp.com:31390 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to kube-01.dev.cmp.com:31390

I am not interested to use the httpsRedirect method (I don’t have http endpoint). Do we have a status about this problem? SNI issue?

I’m on 1.0.6 and encountered the exactly same issue. There’re two workarounds I currently found that make tls work:

1. like @eigokor said, specify httpRedirect and visit http port instead
2. like @themanifold said, specify hosts key as wildcard * and then tls works without redirection