spring-boot: Actuator Istio incorrect management port

Spring boot 2.3.7 Spring Cloud Hoxton SR8 Java 11

We are trying out Istio service mesh but when we deploy the spring boot app actuator endpoint returns incorrect port number in href or no port number.

We have configured management.server.port: 8081 in bootstrap.yaml so we can reach actuator at this port. So naturally when you go to http://10.0.0.1:8081/actuator/ it returns all the end point but with one issue it doesnt include the configured port number. And ofcourse this happens only when Istio side car container is present

{
  "_links": {
    "self": {
      "href": "http://10.0.0.1/actuator",
      "templated": false
    },
    "health-path": {
      "href": "http://10.0.0.1/actuator/health/{*path}",
      "templated": true
    },
    "health": {
      "href": "http://10.0.0.1/actuator/health",
      "templated": false
    },
    "info": {
      "href": "http://10.0.0.1/actuator/info",
      "templated": false
    }
  }
}

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 15 (6 by maintainers)

Most upvoted comments

Thanks for sharing your finding about Undertow, @sabareeshkkanan. Undertow’s ProxyPeerAddressHandler is similar to Reactor Netty’s DefaultHttpForwardedHeaderHandler in that the X-Forwarded-Proto header has no effect on the request’s port.

you can fix it simply by using undertow as webserver as temporary fix

While swapping container may be simple in that it only requires a small change in your pom.xml or build.gradle, it can have quite far-reaching consequences. Anyone else hitting this issue may want to consider setting server.forward-headers-strategy: none as a more focussed workaround that shouldn’t bring any unexpected side-effects with it.

wilkinsona on Jan 14, 2021

As suspected the problem is with the X-Forwarded- headers that Envoy adds and how Tomcat handles them. Here’s an example of the headers in a request as seen by Tomcat:

host = 10.111.130.85:8081
user-agent = curl/7.64.1
accept = */*
x-forwarded-proto = http
x-request-id = e6373443-1668-4fcd-9bc1-7de20e6f4683
x-b3-traceid = df1a38ef71f0cec9b8ce2605691fb12f
x-b3-spanid = b8ce2605691fb12f
x-b3-sampled = 0
content-length = 0

As indicated by the host header, the request has been received on port 8081. Envoy has indicated, via the x-forwarded-proto header, that this is an http request. In the absence of an x-forwarded-port header, Tomcat’s RemoteIPValve assumes that the proxy forwarding the request will use port 80 so this is the port that it configures on the request. There’s an open Envoy issue to add the port header that would tell Tomcat that a non-standard HTTP port is being used.

By contrast, Reactor Netty’s DefaultHttpForwardedHeaderHandler doesn’t change the port when only X-Forwarded-Proto is set. This suits your purposes but would not work for a proxy that expects http to use port 80 by default and https to use port 443 by default.

In this particular case, I think the X-Forwarded-Proto header is causing more harm than good. Without an accompanying X-Forwarded-Port header, the information it provides is potentially misleading and, given that everything is being done over HTTP anyway, the information isn’t adding any value. This means that you can avoid the problem by disabling handling of forward headers by setting server.forward-headers-strategy: none:

$ curl http://10.111.130.85:8081/actuator
{"_links":{"self":{"href":"http://10.111.130.85:8081/actuator","templated":false},"health":{"href":"http://10.111.130.85:8081/actuator/health","templated":false},"health-path":{"href":"http://10.111.130.85:8081/actuator/health/{*path}","templated":true},"info":{"href":"http://10.111.130.85:8081/actuator/info","templated":false}}}

I’m going to close this issue as there’s nothing we can do in Spring Boot to improve the situation. I would recommend subscribing to the Envoy issue so you can keep track of its progress.

wilkinsona on Jan 14, 2021