istio: Incorrect RemoteIP when Authorization Policy is applied to Injected Istio Proxy

I’m not sure if/when that might be supported, but you can approximate it a couple ways:

I think an AuthorizationPolicy like this might work:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: test-authz
spec:
 selector:
   matchLabels:
     app: test
 action: ALLOW
 rules:
 - when:
     - key: request.headers[x-azure-socketip]
       values: ["10.0.0*"]

That just supports basic wildcard matching right now I think, but full regex matching is on the horizon. If you need a full regex, you could also use the VirtualService to filter the traffic with something like this:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: testvs
  labels:
    app: test
spec:
  hosts:
  - test.example.com
  gateways:
  - testgateway.istio-system.svc.cluster.local
  http:
  - match:
    - headers:
        x-azure-socketip:
          regex: '^.*10.0.0.*'
    route:
    - destination:
        host: test
        port:
          number: 80

@rblaine95 Below one worked for me

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: example-chart-policy
  namespace: pf
spec:
  action: ALLOW
  rules:
  - when:
    - key: request.headers[x-forwarded-for]
      values:
      - "xxx.39.121.86*"
      - "xx.206.173.192*"
      - "xxx.236.215.30*"
      - "xx.198.12.81*"
  selector:
    matchLabels:
      app: example-chart

I gave a few things a try there but no joy unfortunately. It seems this stuff is perhaps codified to the gateways? I think the problem with selecting on the IGWs with an AuthorizationPolicy is multi-tenant platforms (like ours) tend to share gateways. Product engineering teams (the platform consumers) don’t get permission to put stuff in the namespace that the IGWs reside.

Quick dump of things tried:

1. Firstly, @yangminzhu’s suggestion of hacking in the annotation at various levels (pod spec, deployment etc.) but didn’t have any luck.

2. Then I tried to get the annotations in place with our mesh deployment (which is istioctlgeneration rather than in-cluster operator). I think the more modern Istio incantation here is something like (please correct me if wrong):

   values:
    sidecarInjectorWebhook:
      injectedAnnotations:
        proxy.istio.io/config: '{"gatewayTopology": { "numTrustedProxies": 1 } }'
   

   This unfortunately doesn’t get accepted by our 1.8 istiod. It crash loops trying to parse the above from YAML to JSON. I tried a few variations but no luck. Perhaps someone else will have more joy.

3. The EnvoyFilter approach

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: mine
  namespace: myns
spec:
  workloadSelector:
    labels:
      app: mine
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': >-
            type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          xff_num_trusted_hops: 1

This got me the furthest. The istioctl listeners objective outlined by @kylebevans is met, but still seems to have no effect, and RBAC debug logging still shows the wrong IP, unlike at the Gateway. Maybe my EnvoyFilter is not sliding in at the right spot?

• key: request.headers[x-forwarded-for] values: - “xxx.39.121.86*” - “xx.206.173.192*” - “xxx.236.215.30*” - “xx.198.12.81*”

Inspired by this suggestion from @anannaya, I tried the following, which worked perfectly:

kind: AuthorizationPolicy
metadata:
  name: my-service-allow-list
  namespace: my-namespace
spec:
  action: ALLOW
  rules:
  - when:
    - key: request.headers[x-envoy-external-address]
      values:
      - 1.2.3.4
      - 1.2.3.5
  selector:
    matchLabels:
      app: my-service 

The only other required change was adding the following annotation to the pod spec of my istio-ingressgateway deployment, following the instructions from the documentation:

proxy.istio.io/config: '{"gatewayTopology" : { "numTrustedProxies": 1 } }'

If numTrustedProxies is set correctly for your topology, then X-Envoy-External-Address will populate with the correct IP from X-Forwarded-For (which can potentially be appended to by each layer 7 proxy along the request path), eliminating the need for a wildcard match.

numTrustedProxies is set to 1 because:

• I have a single reverse proxy upstream of my Istio ingress gateway.
• My cloud load balancer is TCP, not HTTPS, and has no ability to interact with layer 7 headers.

Prior to getting this to work, I tried every conceivable configuration of the from and to fields with no success.

Hope this helps someone.

@pokidov Below configuration at sidecar worked , I have some which routed via ALB So i preferred to apply at the app layer, If you have a all the apps goes via ALB then create Envoyfilter in istio-system namespace.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: xff-envoyfilter-alb-<APPNAME>
  namespace: <NAMESPACE>
spec:
  workloadSelector:
    labels:
      app: <APP LABEL>
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: envoy.filters.network.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            "@type": >-
              type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            xff_num_trusted_hops: 2

@jingslunt I have tried with 1 and 2 as well. Not worked for me.

Hi there, can this issue potentially be re-opened?

We’re experiencing a very similar issue (Client -> AWS ALB -> IGW -> Sidecar -> Workload) where remoteIP is the internal IP address of the AWS ALB, but not Client IP.

Istio version: 1.13.1

Setting this Istio config:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    defaultConfig:
      gatewayTopology:
        numTrustedProxies: 1

Doesn’t add the "xffNumTrustedHops": 1, proxy config, even though the sidecar logs the correct config:

2022-03-07T10:30:17.583866Z	info	Proxy role	ips=[10.98.106.92] type=sidecar id=workload-64d86cf69f-v4zdd.dev-fin domain=my-namespace.svc.cluster.local
2022-03-07T10:30:17.584049Z	info	Apply proxy config from env {"tracing":{"zipkin":{"address":"jaeger-collector.monitoring:9411"},"sampling":50},"gatewayTopology":{"numTrustedProxies":1}}

2022-03-07T10:30:17.590776Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
gatewayTopology:
  numTrustedProxies: 1
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: istio-proxy
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  sampling: 50
  zipkin:
    address: jaeger-collector.monitoring:9411

Adding the annotation to the pods proxy.istio.io/config: '{"gatewayTopology" : { "numTrustedProxies": 1 } }' also doesn’t add the xffNumTrustedHops config when doing istioctl proxy-config listeners -o json -n $NS $POD | grep xff

Example logs from sidecar:

':path', '/'
':method', 'GET'
':scheme', 'https'
'x-forwarded-for', 'IP.ADDR.REDA.CTED,10.98.0.233'
'x-forwarded-proto', 'https'
'x-forwarded-port', '443'
'x-amzn-trace-id', 'Root=1-6225e1e5-4fc1f546078bb09c2b5a52dc'
'cache-control', 'max-age=0'
'upgrade-insecure-requests', '1'
'sec-gpc', '1'
'sec-fetch-site', 'none'
'sec-fetch-mode', 'navigate'
'sec-fetch-user', '?1'
'sec-fetch-dest', 'document'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9'
'x-envoy-external-address', '10.98.0.233'
'x-request-id', '1a66d03f-46c3-4e4c-a8d8-f9cb716740ef'
'x-envoy-peer-metadata', 'lotsamedatada'
'x-envoy-peer-metadata-id', 'router~10.98.123.28~external-igw-7c56b59464-rdmjc.istio-system~istio-system.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', '688c118850d6cae85875fc467aae740f'
'x-b3-spanid', '5875fc467aae740f'
'x-b3-sampled', '0'
'x-envoy-original-path', '/some/workload/'
'x-forwarded-client-cert', 'By=spiffe://cluster.local/ns/my-namespace/sa/workload;Hash=e073a8f25aa7bce4353a81f080856b06bd2b2804bf549e50a15ff2b5ae19d4e7;Subject="";URI=spiffe://cluster.local/ns/istio-system/sa/external-igw-service-account'
, dynamicMetadata: filter_metadata {
  key: "istio_authn"
  value {
  }
}

2022-03-07T10:43:49.088560Z	debug	envoy rbac	enforced denied, matched policy none
2022-03-07T10:43:49.088569Z	debug	envoy http	[C275][S11657379050241880323] Sending local reply with details rbac_access_denied_matched_policy[none]
2022-03-07T10:43:49.088644Z	debug	envoy http	[C275][S11657379050241880323] encoding headers via codec (end_stream=false):
':status', '403'
'content-length', '19'
'content-type', 'text/plain'
'date', 'Mon, 07 Mar 2022 10:43:49 GMT'
'server', 'istio-envoy

After applying the EnvoyFilter detailed at https://github.com/istio/istio/issues/30166#issuecomment-796556188 that selects the workload pods, the xffNumTrustedHops config appears and the remoteIP is the correct Client IP:

2022-03-07T10:50:32.684214Z	debug	envoy rbac	checking request: requestedServerName: outbound_.80_._.workload.my-namespace.svc.cluster.local, sourceIP: 10.98.123.28:36346, directRemoteIP: 10.98.123.28:36346, remoteIP: IP.ADDR.REDA.CTED:0,localAddress: 10.98.117.44:9000, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/external-igw-service-account, dnsSanPeerCertificate: , subjectPeerCertificate: , headers: ':authority', 'example.com'

Side note: Also tried setting gatewayTopology.numTrustedProxies to higher values like 2 and 5 with no luck.

If my understanding of the documentation is correct, this appears like it may be a bug.

FYI, the X-Envoy-External-Address isn’t calculated at the sidecar, and it is not what is used internally by RBAC. It’s calculated at the ingress gateway, and then the sidecars just pass it through among themselves.

Thanks for the very detailed report, that’s a very clear description.

I haven’t looked deep into this but just a quick check, have you tried setting the numTrustedProxies to the sidecar as well?