istio: Unable to remove server header

Without lua:

---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-response-remove-headers
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          server_header_transformation: PASS_THROUGH
  - applyTo: ROUTE_CONFIGURATION
    match:
      context: GATEWAY
    patch:
      operation: MERGE
      value:
        response_headers_to_remove:
        - "x-envoy-upstream-service-time"
        - "server"

What is the status of this? Our secops team wants us to override this for all responses.

@kyessenov Our security/compliance team has an issue with it.

This really needs a much simpler solution for anyone working at scale; probably a single configuration variable.

Most good security/audit teams — and the compliance checks that come along with them — will grade down any servers exposing server as information exposure. As it stands, most users are now exposing implementation data via curl to public clients.

nginx does it with server_tokens off;, apache with ServerSignature Off, etc etc.

In Istio 1.7.2, the following filter configuration worked for me:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  namespace: istio-system
  name: replace-server-name
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            '@type': type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
            server_name: 'undefined'

I’m testing something like:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: server-name-tweaks
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER # http connection manager is a filter in Envoy
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        config:
          server_name: "custom server name"

notice that the example codes on current EnvoyFilter documents does not work on istio 1.3.0 , current MEGER operation only accept config as input.

How can we reopen this issue?

In case if anyone still needs this, I used these filters to remove the server and the x-envoy-upstream-service-time headers (istio 1.9.0)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: response-server-passthrough-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: MERGE
        value:
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
            server_header_transformation: PASS_THROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: response-remove-headers-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |
            function envoy_on_response(response_handle)
              response_handle:headers():remove("x-envoy-upstream-service-time")
              response_handle:headers():remove("server")
            end

How to remove header in latest envoy version, works for me, FYI (istio1.15.5)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ef-removeserver
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER 
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          server_header_transformation: PASS_THROUGH
  - applyTo: ROUTE_CONFIGURATION
    patch:
      operation: MERGE
      value:
        response_headers_to_remove:
        - "x-envoy-upstream-service-time"
        - "server"

Envoy treats some headers as special, including this server header and content-length, and it overrides any plugin (e.g. istio) actions when HTTP specification suggests it. For example, empty bodies are going to have content length 0 because that’s what clients expect.

Do you have an issue with the server header itself, or the value that the header is set to?

It is a form of CWE-200: Information Exposure (ref: https://cwe.mitre.org/data/definitions/200.html)

This type of leaking of technology choices in HTTP responses allows threat actors to enumerate target environments using certain tools and versions and target them when vulnerabilities are identified. This is why both internal SecOps teams and external pen testers will routinely flag Server banners that include tech choices and version numbers as low to medium issues for engineering teams to address.

How to change server in latest envoy version, works for me. FYI (istio1.15.5)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ef-removeserver
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER 
    match:
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          server_name: "My Server"

@httran13 I just tried your envoyfilter on 1.8.3 and it’s working for me:

$ curl -I 169.62.142.131
HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
date: Tue, 02 Mar 2021 00:42:05 GMT
content-length: 19
x-envoy-upstream-service-time: 4
server: istio-envoy

$ k apply -f ./remove-header.yaml
envoyfilter.networking.istio.io/replace-server-name created

$ curl -I 169.62.142.131
HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
date: Tue, 02 Mar 2021 00:42:31 GMT
content-length: 19
x-envoy-upstream-service-time: 5

Should server and uptime be removed by default with opt-in?

If we launched Istio 1.0 today, yes. At this point making backwards incompat changes like this is hard though. Ambient will probably default to this.

@nrjpoddar good news! thx guys!

We discussed this during Feb 11 2021 Networking WG meeting and agreed on configuring Envoy to always set Server header value as PASS_THROUGH instead of the currently configured override value.

@nrjpoddar I think it will be enough for istio to setting server_header_transformation to PASS_THROUGH by default. Then we will add, remove, or override any headers with VirtualService if needed or passing headers through from applications.

Thanks @jsolv Why is the VS needed? Isn’t the passthrough settings alone sufficient?

Just to clarify patch by @softcane above… this does not actually guarantee removal of the server header; what it does do is it stops envoy from overwriting the server header that came back from upstream, hence the phrase PASS_THROUGH.

Also note that from Envoy 1.7.0 it looks like these headers can be suppressed after all… https://www.envoyproxy.io/docs/envoy/v1.7.0/api-v2/config/filter/http/router/v2/router.proto

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: remove-server-header-filter
  namespace: test
spec:
  workloadSelector:
    labels:
      service.istio.io/canonical-name: 
istio-ingressgateway
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.http_connection_manager"
      patch:
        operation: MERGE
        value:
          config:
            server_header_transformation: PASS_THROUGH

This envoy filter will remove the server header in the response.

@pingz Thank you for the solution. It does work on Istio 1.3.4. P.S. Please add the apiVersion field to your yaml example.