spring-boot: Duplicate "Transfer-Encoding: chunked" header

I am getting duplicate Transfer-Encoding: chunked headers, which causes my Nginx Ingress Controller to return 502. Weird thing about this: it starts happening randomly after the spring boot instance runs for some time. When the specific pod is restarted, it works fine again.

Full error message from Nginx Ingress controller:

2023/10/01 19:00:12 [error] 1275#1275: *631585 upstream sent duplicate header line: "Transfer-Encoding: chunked", previous value: "Transfer-Encoding: chunked" while reading response header from upstream, client: xxx, server: xxx, request: "GET /v2/projects/export?languages=de&format=JSON&zip=false HTTP/1.1", upstream: "http://10.224.1.48:8080/v2/projects/export?languages=de&format=JSON&zip=false", host: "app.tolgee.io"
194.74.103.210 - - [01/Oct/2023:18:59:47 +0000] "GET /v2/projects/xxx/translations/en HTTP/1.1" 502 559 "http://localhost:5173/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36" "-"   

image

The sensitive handler method 1 code. The sensitive handler method 2 code

I am not sure whether it only happens to these 2 endpoints. They are executed very frequently and have bigger responses.

SpringBoot version: 2.7.13 Nginx.com Ingress version: 0.18.1

(I don’t think it’s related to this), since I don’t work with response entities returned from http client.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 22 (5 by maintainers)

Most upvoted comments

posting this in a couple spring issues on this topic in the hopes it helps others. the best writeup of this problem i’ve encountered is https://knowledge.broadcom.com/external/article/298108/too-many-transfer-encodings-502-chunked.html. Trigger number 3 in that article was our problem

In Spring-based microservice applications, when using RestTemplate from a Controller annotated class and returning the result of RestTemplate.exchange in a method that returns a ResponseEntity. Similar to situation 1, this will cause a problem if the microservice to which you are talking returns a transfer-encoding header because RestTemplate.exchange will copy all of the response headers from the client response into the ResponseEntity it creates, including the transfer-encoding header.

@Hollerweger 's comment in https://github.com/spring-projects/spring-framework/issues/21523#issuecomment-1233264593 provides an alternative that avoided this problem for us.

For most usecases creating a new ResponseEntity should solve the issue (when original Headers are not required at all): return ResponseEntity.status(responseEntity.getStatusCode()).body(responseEntity.getBody())

I’m also running into this issue, my webserver is using tomcat; In the process of debugging, it was found that when tomcat processed the response of http1.1, a piece of code added duplicate chunked, which did not check whether there was chunked in the current head; This will result in duplicate chunked rows; The specific code location is: org.apache.coyote.http11.Http11Processor#prepareResponse Image

Hope it helps;

@jakub-blazej-telekom I believe this issue comment is relevant. Your web endpoint should never add this header in the first place, as only the web server is in charge of such headers. If your application is proxying calls to a another REST endpoint, it should clean up the response headers before forwarding them (that’s what proxies do). This applies not only to Transfer-Encoding but also security headers.

sorry, I missed that. Here you go: header-demo.zip commenting the header line out removes “both” headers. Tried this on spring-boot 3.2.5 and 3.3.0

@GetMapping("/")
@ResponseBody
public ResponseEntity<String> getMain() {
    HttpHeaders headers = new HttpHeaders();
    headers.set(HttpHeaders.TRANSFER_ENCODING, "chunked");

    return ResponseEntity.ok()
            .headers(headers)
            .body("hello");
}

image

Hi @sky-philipalmeida , unfortunately yes, but it’s not happening that often. Before, we’ve encountered this and got to restart the affected node around 10x a week. After upgrading to Spring Boot 3 it happened just once in ~2 months.

Perhaps you could add a Filter, positioned first in the chain, that checks the response headers after calling doFilter. If there are no Transfer-Encoding headers in the response at this point, it would suggest that the web server is setting the header twice. If there are one or more Transfer-Encoding headers, that would suggest it’s Spring or application code that is setting the header.