istio: serviceentry / virtual service not work when target to outside server.

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

I need let ingress gateway as a proxy, for outside domain. It works for http://httpbin.org, but cannot work for https://. (mtls is set strict in cluster)

Happy case

# configure
kubectl replace --force -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-ext
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
spec:
  gateways:
  - istio-system/apigee-gateway
  hosts:
  - httpbin-onecloudsol-dev.api.dev-sol-us.company.io
  http:
  - timeout: 3s
    route:
    - destination:
        host: httpbin.org
      weight: 100
EOF

# Test
$ curl https://httpbin-onecloudsol-dev.api.dev-sol-us.company.io/ip
{
  "origin": "192.x.x.x,10.212.186.101, 54.148.123.253"
}

# clean up
k delete se httpbin-ext
k delete vs  httpbin-ext

Fail case 1 for https:

# configure
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 443
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
spec:
  gateways:
  - istio-system/apigee-gateway
  hosts:
  - httpbin443-onecloudsol-dev.api.dev-sol-us.company.io
  http:
  - timeout: 3s
    route:
    - destination:
        host: httpbin.org
      weight: 100
EOF

# Test
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

# delete DestinationRule and test again
$ k delete dr originate-tls
destinationrule.networking.istio.io "originate-tls" deleted
 
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
</body>
</html>

# clean up all
k delete se httpbin
k delete vs httpbin-ext

Fail case 2 for https:

# configure
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
spec:
  gateways:
  - istio-system/apigee-gateway
  hosts:
  - httpbin443-onecloudsol-dev.api.dev-sol-us.company.io
  http:
  - timeout: 3s
    route:
    - destination:
        host: httpbin.org
      weight: 100
EOF

# Test
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

# delete DestinationRule and test again
$ k delete dr originate-tls
destinationrule.networking.istio.io "originate-tls" deleted
 
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
</body>
</html>

# clean up all
k delete se httpbin
k delete vs httpbin-ext

Version

$ istio verison 1.20
$ kubernetes version: 1.27

Additional Information

mtls is enabled with strict

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 30 (28 by maintainers)

Most upvoted comments

@johnzheng1975 Could you please try with what John Howard mentioned above to have the hardcoded SNI in DestinationRule? If that works, you don’t need turn on insecureSkipVerify

My team is doing exactly what you are trying to achieve and we have hardcoded SNI in all external endpoints’ destination rule. It works well from Istio 1.15 to Istio 1.19. We didn’t maintain any clusters with Istio >= 1.20 yet though.

Full reproducer:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin
  namespace: istio-system
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 443
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
  namespace: istio-system
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
      sni: httpbin.org # DOES FIX IT
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
  namespace: istio-system
spec:
  gateways:
  - istio-system/web-gateway
  hosts:
  - httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io
  http:
  - timeout: 3s
    rewrite:
      authority: httpbin.org # DOES NOT FIX IT
    route:
    - destination:
        host: httpbin.org
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: web-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---

A few questions here…

  • Do we really need auto_sni when we have a FQDN DR? Why not just hardcode sni to httpbin.org here?
  • Does rewrite.authority in route not impact auto_sni?

cc @kfaseela @jacob-delgado

Previoulsy we do not verify, this is now changed by VERIFY_CERTIFICATE_AT_CLIENT, you can tune it to false, i believe it can also work foryou, so you donot need a envoyfilter to do that

are you really using httpbin in your tests or something else? I don’t see why httpbin would fail to verify…

I doubt if this is caused by VERIFY_CERTIFICATE_AT_CLIENT default to true now. Or try set InsecureSkipVerify = false

Egress gateway is not a must, actually ingress gateway Can act as egress gateway too if you want.

i can not even searched outbound|80||httpbin.org in yout your configdum, and not even a dynamic listeners there?

what is the gateway created and xds configuration generated?