istio: VM mesh expansion, ISTIO_INBOUND_PORTS conflicts with Docker container outbound traffic

Bug description No outbound traffic is possible from Docker containers on ports that are defined in ISTIO_INBOUND_PORTS.

Use case: running Docker containers on mesh expansion VMs.

In our case we are running an application that listens on port 80 but also has to reach out for the GCE metadata service (also runs on Port 80) to get the user token of the VM service account. This is the default behaviour doing authentication for most GCP related service libraries.

It seems that this outbound traffic interferes with ISTIO_INBOUND_PORTS when the same port is used for egress. Doing outbound calls works fine when directly done on the VM but not from the containers, I would suspect that the additional iptables rules needed for docker are not reflected in the Istio iptables rule set or some configuration is missing from our side.

This behaviour is reproducible with any external service running on ports that are also defined in ISTIO_INBOUND_PORTS. Currently it looks like that outbound traffic on these ports gets redirected back to the origin.

This setup was definitely working until Istio 1.5.1

[ ] Docs [ ] Installation [ x] Networking [ ] Performance and Scalability [ ] Extensions and Telemetry [ ] Security [ ] Test and Release [x ] User Experience [ ] Developer Infrastructure

Expected behavior Outbound traffic from containers does not get affected by ISTIO_INBOUND_PORTS settings.

Steps to reproduce the bug

• Setup a VM expanding the mesh
• Set ISTIO_INBOUND_PORTS=80
• Do a curl directly from the VM => that should succeed

# curl -v http://whatismyip.akamai.com
* Rebuilt URL to: http://whatismyip.akamai.com/
*   Trying 88.221.25.251...
* TCP_NODELAY set
* Connected to whatismyip.akamai.com (88.221.25.251) port 80 (#0)
> GET / HTTP/1.1
> Host: whatismyip.akamai.com
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK

• Do a curl from a docker container on the VM => that will fail

# docker run --rm -it curlimages/curl -v http://whatismyip.akamai.com
*   Trying 88.221.25.251:80...
* Connected to whatismyip.akamai.com (88.221.25.251) port 80 (#0)
> GET / HTTP/1.1
> Host: whatismyip.akamai.com
> User-Agent: curl/7.72.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Mon, 05 Oct 2020 11:16:48 GMT
< server: istio-envoy
< x-envoy-decorator-operation: gitlab.gitlab-omnibus.svc.cluster.local:80/*
< 
* Connection #0 to host whatismyip.akamai.com left intact

• change ISTIO_INBOUND_PORTS to a non conflicting port

ISTIO_INBOUND_PORTS=8080

• restart the istio-sidecar

systemctl restart istio

• retry the curl from the container => should work this time

# docker run --rm -it curlimages/curl -v http://whatismyip.akamai.com
*   Trying 88.221.144.73:80...
* Connected to whatismyip.akamai.com (88.221.144.73) port 80 (#0)
> GET / HTTP/1.1
> Host: whatismyip.akamai.com
> User-Agent: curl/7.72.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/html
< Content-Length: 12
< Expires: Mon, 05 Oct 2020 11:20:20 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Mon, 05 Oct 2020 11:20:20 GMT
< Connection: keep-alive

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)

client version: 1.7.3
control plane version: 1.7.3
data plane version: 1.7.3 (4 proxies), 1.7.0 (3 proxies)

Client Version: v1.19.2
Server Version: v1.16.13-gke.401

How was Istio installed?

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      meshExpansion:
        enabled: true

Environment where bug was observed (cloud vendor, OS, etc) GKE, GCE (ubuntu-minimal-1804-bionic-v20200520)

debug log error case with colliding ports The 127.0.0.1 and the match with the inbound is confusing

2020-10-05T11:32:32.274711Z     debug   envoy filter    original_dst: New connection accepted
2020-10-05T11:32:32.274768Z     debug   envoy filter    tls inspector: new connection accepted
2020-10-05T11:32:32.275324Z     debug   envoy filter    http inspector: new connection accepted
2020-10-05T11:32:32.275440Z     debug   envoy conn_handler      [C2] new connection
2020-10-05T11:32:32.275554Z     debug   envoy http      [C2] new stream
2020-10-05T11:32:32.275615Z     debug   envoy http      [C2][S8257163499414696093] request headers complete (end_stream=true):
':authority', 'whatismyip.akamai.com'
':path', '/'
':method', 'GET'
'user-agent', 'curl/7.72.0-DEV'
'accept', '*/*'

2020-10-05T11:32:32.275623Z     debug   envoy http      [C2][S8257163499414696093] request end stream
2020-10-05T11:32:32.275763Z     debug   envoy filter    AuthenticationFilter::decodeHeaders with config
policy {
  peers {
    mtls {
      mode: PERMISSIVE
    }
  }
}
skip_validate_trust_domain: true

2020-10-05T11:32:32.275776Z     debug   envoy filter    [C2] validateX509 mode PERMISSIVE: ssl=false, has_user=false
2020-10-05T11:32:32.275781Z     debug   envoy filter    Payload has not peer authentication data
2020-10-05T11:32:32.275787Z     debug   envoy filter    Set principal from peer:
2020-10-05T11:32:32.275792Z     debug   envoy filter    Origin authenticator succeeded
2020-10-05T11:32:32.275802Z     debug   envoy filter    Saved Dynamic Metadata:

2020-10-05T11:32:32.275823Z     debug   envoy router    [C2][S8257163499414696093] cluster 'inbound|80|http-omnibus|gitlab.gitlab-omnibus.svc.cluster.local' match for URL '/'
2020-10-05T11:32:32.275886Z     debug   envoy router    [C2][S8257163499414696093] router decoding headers:
':authority', 'whatismyip.akamai.com'
':path', '/'
':method', 'GET'
':scheme', 'http'
'user-agent', 'curl/7.72.0-DEV'
'accept', '*/*'
'x-forwarded-proto', 'http'
'x-request-id', 'ad3ca040-3ba4-459d-aaf7-0109c1602ddf'
'x-b3-traceid', 'ae769085a794d03e15b364c0de615bb1'
'x-b3-spanid', '15b364c0de615bb1'
'x-b3-sampled', '0'

2020-10-05T11:32:32.275899Z     debug   envoy pool      queueing request due to no available connections
2020-10-05T11:32:32.275906Z     debug   envoy pool      creating a new connection
2020-10-05T11:32:32.275953Z     debug   envoy client    [C3] connecting
2020-10-05T11:32:32.275960Z     debug   envoy connection        [C3] connecting to 127.0.0.1:80
2020-10-05T11:32:32.276015Z     debug   envoy connection        [C3] connection in progress
2020-10-05T11:32:32.276070Z     debug   envoy connection        [C3] delayed connection error: 111
2020-10-05T11:32:32.276077Z     debug   envoy connection        [C3] closing socket: 0
2020-10-05T11:32:32.276094Z     debug   envoy client    [C3] disconnect. resetting 0 pending requests
2020-10-05T11:32:32.276102Z     debug   envoy pool      [C3] client disconnected, failure reason:
2020-10-05T11:32:32.276113Z     debug   envoy router    [C2][S8257163499414696093] upstream reset: reset reason connection failure
2020-10-05T11:32:32.276172Z     debug   envoy http      [C2][S8257163499414696093] Sending local reply with details upstream_reset_before_response_started{connection failure}
2020-10-05T11:32:32.276217Z     debug   envoy http      [C2][S8257163499414696093] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '91'
'content-type', 'text/plain'
'date', 'Mon, 05 Oct 2020 11:32:31 GMT'
'server', 'istio-envoy'
'x-envoy-decorator-operation', 'gitlab.gitlab-omnibus.svc.cluster.local:80/*'

# iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 1 packets, 67 bytes)
 pkts bytes target     prot opt in     out     source               destination         
49221 2947K DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
    1    60 ISTIO_INBOUND  tcp  --  any    any     anywhere             anywhere            

Chain INPUT (policy ACCEPT 1 packets, 60 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 49 packets, 3789 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  any    any     anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL
   10   600 ISTIO_OUTPUT  tcp  --  any    any     anywhere             anywhere            

Chain POSTROUTING (policy ACCEPT 50 packets, 3849 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  626 40773 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere            
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.2           172.17.0.2           tcp dpt:https
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.2           172.17.0.2           tcp dpt:http
    0     0 MASQUERADE  tcp  --  any    any     172.17.0.2           172.17.0.2           tcp dpt:ssh

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere            
    0     0 DNAT       tcp  --  !docker0 any     anywhere             gitlab-omnibus.c.rd-k8s-common.internal  tcp dpt:https to:172.17.0.2:443
  273 16380 DNAT       tcp  --  !docker0 any     anywhere             gitlab-omnibus.c.rd-k8s-common.internal  tcp dpt:http to:172.17.0.2:80
 4629  278K DNAT       tcp  --  !docker0 any     anywhere             gitlab-omnibus.c.rd-k8s-common.internal  tcp dpt:2222 to:172.17.0.2:22

Chain ISTIO_INBOUND (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15008
    1    60 ISTIO_IN_REDIRECT  tcp  --  any    any     anywhere             anywhere             tcp dpt:http
    0     0 ISTIO_IN_REDIRECT  tcp  --  any    any     anywhere             anywhere             tcp dpt:2222

Chain ISTIO_IN_REDIRECT (6 references)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15006

Chain ISTIO_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  any    lo      localhost            anywhere            
    0     0 ISTIO_IN_REDIRECT  all  --  any    lo      anywhere            !localhost            owner UID match istio-proxy
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner UID match istio-proxy
    9   540 RETURN     all  --  any    any     anywhere             anywhere             owner UID match istio-proxy
    0     0 ISTIO_IN_REDIRECT  all  --  any    lo      anywhere            !localhost            owner UID match root
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner UID match root
    1    60 RETURN     all  --  any    any     anywhere             anywhere             owner UID match root
    0     0 ISTIO_IN_REDIRECT  all  --  any    lo      anywhere            !localhost            owner GID match admin
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner GID match admin
    0     0 RETURN     all  --  any    any     anywhere             anywhere             owner GID match admin
    0     0 ISTIO_IN_REDIRECT  all  --  any    lo      anywhere            !localhost            owner GID match root
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner GID match root
    0     0 RETURN     all  --  any    any     anywhere             anywhere             owner GID match root
    0     0 RETURN     all  --  any    any     anywhere             localhost           
    0     0 ISTIO_REDIRECT  all  --  any    any     anywhere             10.12.48.0/20       
    0     0 RETURN     all  --  any    any     anywhere             anywhere            

Chain ISTIO_REDIRECT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15001