istio: IP no longer allowed in ServiceEntry host field

Describe the bug With the 1.1 release, the host field of a service entry can no longer be an IP address. Perhaps this is intended, but if so, additional documentation needs to be added on how to allow an IP address for a service entry.

Expected behavior An IP address is allowed for a service entry. Our mesh is configured with REGISTRY_ONLY and our application needs to access the AWS ec2 metadata endpoint at 169.254.169.254. Have tried

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: aws-metadata
spec:
  hosts:
  - ec2-metadata.local
  endpoints:
  - address: 169.254.169.254
  ports:
  - number: 80
    name: http-aws-metadata
    protocol: HTTP
  resolution: STATIC
  location: MESH_EXTERNAL

as a hopeful workaround, but this doesnt work either.

Steps to reproduce the bug Unable to create the following service entry which worked in istio 1.0.6

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: aws-metadata
spec:
  hosts:
  -169.254.169.254
  ports:
  - number: 80
    name: http-aws-metadata
    protocol: HTTP

Version

client version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
citadel version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
galley version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
galley version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
ilbgateway version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
ilbgateway version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
ingressgateway version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
ingressgateway version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.0-17-g2b13318"}
nodeagent version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
nodeagent version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
nodeagent version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
nodeagent version: version.BuildInfo{Version:"", GitRevision:"", User:"", Host:"", GolangVersion:"", DockerHub:"", BuildStatus:"", GitTag:""}
pilot version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
pilot version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
policy version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
policy version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
sidecar-injector version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
telemetry version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}
telemetry version: version.BuildInfo{Version:"1.1.1", GitRevision:"2b1331886076df103179e3da5dc9077fed59c989-dirty", User:"root", Host:"7077232d-4c6c-11e9-813c-0a580a2c0506", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Modified", GitTag:"1.1.0-17-g2b13318"}

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.7", GitCommit:"65ecaf0671341311ce6aea0edab46ee69f65d59e", GitTreeState:"clean", BuildDate:"2019-01-24T19:22:45Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}

Installation via helm template

Environment AWS

Cluster state Will provide if necessary, but dump script doesnt work with multiple pilot pods.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 2
Comments: 34 (12 by maintainers)

Most upvoted comments

@frankbu the x forwarded for header was changed in 1.2.0 which causes this issue, Here is the PR with merge resolved. #15167 reverts it which fixes the issue. I tested this by manual curling with x forwarded for (not through envoy) and it fails

Google support mentioned a fix is in preparation for 1.2.2 and being tested. Now i don’t know if the fix means revert. Just FYI Having an example from istio team is required now, we are all losing time deploying istio on GKE and using GCP services because of the lack of examples and guidance from the Google/Istio team. So it will be highly appreciated to have a recommended approach for that metadata access

Jonathan34 on Jun 28, 2019

i managed to get it working using the below and I don’t get 403 anymore. The problem seemsm to be the x-forwarded-for.

PS: i am not using workload identity and the below are deployed in the same namespace as my workloads

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: metadata-google-internal
spec:
  hosts:
    - metadata.google.internal
  http:
    - route:
        - destination:
            host: metadata.google.internal
          headers:
            request:
             remove:
                - x-forwarded-for
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: metadata-google-internal
spec:
  hosts:
    - metadata.google.internal
  addresses:
    - 169.254.169.254
  ports:
    - name: http
      number: 80
      protocol: HTTP
    - name: https
      number: 443
      protocol: HTTPS
  location: MESH_INTERNAL
  resolution: DNS

Jonathan34 on Jun 28, 2019

I still haven’t found a permutation that works in 1.2.0. Previously, was using this in GKE on Istio up to 1.1.8:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: gce-metadata
spec:
  hosts:
  - metadata.google.internal # not used with addresses
  addresses:
  - 169.254.169.254
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: STATIC
  location: MESH_EXTERNAL
  endpoints:
  - address: 169.254.169.254

That still passes the validation, but doesn’t work. I’ve tried appending a /32 to the addresses: line, but that apparently gets evaluated in a way that the validator thinks there’s no change and discards the edit. I added a suffix to the port name to force a change, but I still get 403 in the application container. Dropping the endpoints attribute and changing resolution to NONE also doesn’t work. Putting the endpoints back and changing resolution to DNS also didn’t work. I’m sure that the endpoint is available because if I exec into the istio-proxy container instead, I can curl it just fine.

(Note that this is for a freshly-deployed cluster, not upgraded from 1.1.x.)

Does anyone have something that actually works with Istio 1.2.0? And could we get that added as an example to the ServiceEntry page since that’s going to be a very popular use case and would work in both GKE and AWS EC2?

kbruner on Jun 25, 2019

As of 1.4.5, the ServiceEntry below allows access for both http://169.254.169.254 and http://metadata.google.internal

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: gke-metadata-server
spec:
  hosts:
    - metadata.google.internal
  addresses:
    - 169.254.169.254
  ports:
    - name: http
      number: 80
      protocol: http
  resolution: STATIC
  location: MESH_EXTERNAL
  endpoints:
    - address: 169.254.169.254

cagataygurturk on Mar 3, 2020

Seeing this issue in 1.2 also:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: aws-metadata
spec:
  endpoints:
    - address: 169.254.169.254
  hosts:
    - 169.254.169.254
  location: MESH_EXTERNAL
  ports:
    - name: http
      number: 80
      protocol: HTTP
  resolution: STATIC

chrisob on Jun 19, 2019