reactor-netty: WebClient in Tomcat on Kubernetes Memory Leak

We’re trying to use WebClient in our microservice to get large data object from external backend service, and the microservice runs on Kubernetes cluster (I tested on both Docker Desktop Kubernetes or Azure Kubernetes). We’re seeing the pod memory go up until the pod is killed and restarted

Expected Behavior

The pod should not be killed, and the memory usage should be similar or less than using the RestTemplate.

Actual Behavior

When we make external service calls using Netty’s WebClient, we’re seeing the memory increases seemingly without limit. I used JCMD to check GC and get the heapdump. GC seems normal and there is no leak found from the heapdump. If I run the microservice as a standalone app, I confirmed the GC happened with VisualVM.

When I run the microservices on Docker Desktop Kubernetes, the memory is consistently going up until the instance is eventually killed without any outOfMemory errors.

Steps to Reproduce

I’ve created sample projects for all the things I tried. The code is in test-service-client. It has 3 branches.

The backend: test-backend branch. For testing purpose, I simplify the result that has about 8mb, 16mb, 24mb and 32mb bogus data. The actual external service returns a big object that contains different information. You can run “kubectl apply -f test-backend.yaml” to create the pod. The Tomcat WebClient: test-webclient-tomcat branch. The microservice runs with -Xmx250M. You can run “kubectl apply -f test-webclient-tomcat.yaml” to create the pod. The Tomcat RestTemplate: test-rest-template branch. The microservice runs with -Xmx250M. You can run “kubectl apply -f test-rest-template.yaml” to create the pod.

After you checkout the branch, you can run “gradle docker” to build the docker image and deploy the docker image to any kubernetes cluster. The pod memory resource is set to 320M. You can monitor the memory usage using Kubernetes Dashboard or “kubectl describe podMetrics <pod name>”.

After the pods are deployed and port forward to a local port, I trigger the Tomcat WebClient service one at a time. The pod will be killed or restarted after 4 or 6 requests are completed.

@Service
public class DataServiceImpl implements DataService {

  private static final Logger log = LoggerFactory.getLogger(DataServiceImpl.class);

  @Value("${app.source.url}")
  private String sourceUrl;

  @Autowired private WebClient webClient;

  public DataResult getData(String imageNo) {
    try {
      var httpHeaders = new HashMap<String, String>();
      var responseStr =
          webClient
              .get()
              .uri(sourceUrl + (StringUtils.hasText(imageNo) ? "/" + imageNo : ""))
              .headers(headers -> headers.setAll(httpHeaders))
              .retrieve()
              .bodyToMono(String.class)
              .block();

      if (StringUtils.hasText(responseStr)) {
        var result = new DataResult();
        result.setCode(DataResultCode.OK);
        result.setData(responseStr);
        return result;
      }
    } catch (Exception e) {
      log.error("Failed...", e);
      var result = new DataResult();
      result.setCode(DataResultCode.ERROR);
      result.setMessage(e.getMessage());
      return result;
    }
    return null;
  }
}

Your Environment

• Reactor version(s) used: Spring Webflux 5.3.22, Reactor Core 3.4.23
• Reactor Netty version: 1.0.23
• JVM version (java -version): OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1 (build 11.0.16.1+1, mixed mode)
• OS and version (eg. uname -a): Docker image adoptopenjdk/openjdk11