istio: serviceentry / virtual service not work when target to outside server.
Is this the right place to submit this?
- This is not a security vulnerability or a crashing bug
- This is not a question about how to use Istio
Bug Description
I need let ingress gateway as a proxy, for outside domain. It works for http://httpbin.org, but cannot work for https://. (mtls is set strict in cluster)
Happy case
# configure
kubectl replace --force -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: httpbin-ext
spec:
hosts:
- httpbin.org
ports:
- number: 80
name: http
protocol: HTTP
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin-ext
spec:
gateways:
- istio-system/apigee-gateway
hosts:
- httpbin-onecloudsol-dev.api.dev-sol-us.company.io
http:
- timeout: 3s
route:
- destination:
host: httpbin.org
weight: 100
EOF
# Test
$ curl https://httpbin-onecloudsol-dev.api.dev-sol-us.company.io/ip
{
"origin": "192.x.x.x,10.212.186.101, 54.148.123.253"
}
# clean up
k delete se httpbin-ext
k delete vs httpbin-ext
Fail case 1 for https:
# configure
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: httpbin
spec:
hosts:
- httpbin.org
ports:
- number: 80
name: http
protocol: HTTP
targetPort: 443
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: originate-tls
spec:
host: httpbin.org
trafficPolicy:
tls:
mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin-ext
spec:
gateways:
- istio-system/apigee-gateway
hosts:
- httpbin443-onecloudsol-dev.api.dev-sol-us.company.io
http:
- timeout: 3s
route:
- destination:
host: httpbin.org
weight: 100
EOF
# Test
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end
# delete DestinationRule and test again
$ k delete dr originate-tls
destinationrule.networking.istio.io "originate-tls" deleted
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
</body>
</html>
# clean up all
k delete se httpbin
k delete vs httpbin-ext
Fail case 2 for https:
# configure
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: httpbin
spec:
hosts:
- httpbin.org
ports:
- number: 443
name: https
protocol: HTTPS
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: originate-tls
spec:
host: httpbin.org
trafficPolicy:
tls:
mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin-ext
spec:
gateways:
- istio-system/apigee-gateway
hosts:
- httpbin443-onecloudsol-dev.api.dev-sol-us.company.io
http:
- timeout: 3s
route:
- destination:
host: httpbin.org
weight: 100
EOF
# Test
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end
# delete DestinationRule and test again
$ k delete dr originate-tls
destinationrule.networking.istio.io "originate-tls" deleted
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
</body>
</html>
# clean up all
k delete se httpbin
k delete vs httpbin-ext
Version
$ istio verison 1.20
$ kubernetes version: 1.27
Additional Information
mtls is enabled with strict
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 30 (28 by maintainers)
@johnzheng1975 Could you please try with what John Howard mentioned above to have the hardcoded SNI in DestinationRule? If that works, you don’t need turn on
insecureSkipVerifyMy team is doing exactly what you are trying to achieve and we have hardcoded SNI in all external endpoints’ destination rule. It works well from Istio 1.15 to Istio 1.19. We didn’t maintain any clusters with Istio >= 1.20 yet though.
Full reproducer:
A few questions here…
cc @kfaseela @jacob-delgado
Previoulsy we do not verify, this is now changed by VERIFY_CERTIFICATE_AT_CLIENT, you can tune it to false, i believe it can also work foryou, so you donot need a envoyfilter to do that
are you really using httpbin in your tests or something else? I don’t see why httpbin would fail to verify…
I doubt if this is caused by
VERIFY_CERTIFICATE_AT_CLIENTdefault to true now. Or try set InsecureSkipVerify = falseEgress gateway is not a must, actually ingress gateway Can act as egress gateway too if you want.
i can not even searched
outbound|80||httpbin.orgin yout your configdum, and not even a dynamic listeners there?what is the gateway created and xds configuration generated?