spring-boot: Spring Boot v3.4.0 causes our staging & production environment to hang and time out
Our services work on all Spring Boot versions prior to v3.4.0 and have been for years. v3.4.0 works in our dev environment and we are unable to reproduce what is occurring in staging and production.
We have 4 Spring Boot apps running on GCP
- Spring Authorization server (does not seem to be affected)
- Spring Boot with webflux doesn’t seem to be affected
- Two Spring Boot Web MVC apps are affected
Here is what happens once the deployment hits staging / production:
- No errors in the startup log
- Very quickly, if not immediately, after deployment, requested endpoint starts to hang
GCP metrics (response times goes to 300 seconds and times out after release)
MongoDB console (Activity decreases after release)
Cloud Run startup log that shows Spring Boot starting up error free and goes directly to timing out
Maven dependency tree: dependencies.txt.zip
The next step for us would be to turn on logging to high to see anything interesting shows up. Just spent a Saturday afternoon rolling back from production and trying to figure out where it was coming from. At this moment, I am completely clueless.
Any help would be appreciated.
We deploy Spring Boot with mvn spring-boot:build-image and the plugin config looks like this:
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<id>process-aot</id>
<configuration>
<profiles>local,staging,prod</profiles>
</configuration>
</execution>
</executions>
<configuration>
<image>
<env>
<BPE_DELIM_JAVA_TOOL_OPTIONS xml:space="preserve"> </BPE_DELIM_JAVA_TOOL_OPTIONS>
<BPE_APPEND_JAVA_TOOL_OPTIONS>--add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -Dspring.profiles.active=${environment} -Djava.security.manager=allow</BPE_APPEND_JAVA_TOOL_OPTIONS>
<BP_JVM_TYPE>JRE</BP_JVM_TYPE>
<BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
</env>
</image>
</configuration>
</plugin>
Development environment: Mac OS Sequoia Atlas CLI MongoDb 8 Java 23
Providers: MongoDb Atlas MongoDb 8 Google Cloud Platform / Cloud run Java 23
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 25 (12 by maintainers)
I think you should engage Google support at this point. The goal being to understand why a server that appears to be ready and able to accept requests is unable to do so. Perhaps they can provide some diagnostics about the network.
If it’s not Mongo that’s the problem, then I would turn attention to the web server. Looking at the thread dump, you appear to be using Undertow (the XNIO threads). Have you tried with Tomcat or Jetty? If the problem doesn’t occur when using WebFlux (and assuming that you’re using the default of Netty there), that may be another pointer towards an Undertow problem.