spring-boot: Spring Boot v3.4.0 causes our staging & production environment to hang and time out

Our services work on all Spring Boot versions prior to v3.4.0 and have been for years. v3.4.0 works in our dev environment and we are unable to reproduce what is occurring in staging and production.

We have 4 Spring Boot apps running on GCP

  • Spring Authorization server (does not seem to be affected)
  • Spring Boot with webflux doesn’t seem to be affected
  • Two Spring Boot Web MVC apps are affected

Here is what happens once the deployment hits staging / production:

  • No errors in the startup log
  • Very quickly, if not immediately, after deployment, requested endpoint starts to hang

GCP metrics (response times goes to 300 seconds and times out after release) Screenshot 2024-11-30 at 15 15 56

MongoDB console (Activity decreases after release) screencapture-cloud-mongodb-v2-590aeb37c0c6e35c7ce2d87f-2024-11-30-15_09_07

Cloud Run startup log that shows Spring Boot starting up error free and goes directly to timing out Screenshot 2024-11-30 at 19 47 40

Maven dependency tree: dependencies.txt.zip

The next step for us would be to turn on logging to high to see anything interesting shows up. Just spent a Saturday afternoon rolling back from production and trying to figure out where it was coming from. At this moment, I am completely clueless.

Any help would be appreciated.

We deploy Spring Boot with mvn spring-boot:build-image and the plugin config looks like this:

<plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>process-aot</id>
                        <configuration>
                            <profiles>local,staging,prod</profiles>
                        </configuration>
                    </execution>
                </executions>
                <configuration>
                    <image>
                        <env>
                            <BPE_DELIM_JAVA_TOOL_OPTIONS xml:space="preserve"> </BPE_DELIM_JAVA_TOOL_OPTIONS>
                            <BPE_APPEND_JAVA_TOOL_OPTIONS>--add-opens=java.base/java.time=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -Dspring.profiles.active=${environment} -Djava.security.manager=allow</BPE_APPEND_JAVA_TOOL_OPTIONS>
                            <BP_JVM_TYPE>JRE</BP_JVM_TYPE>
                            <BP_JVM_VERSION>${java.version}</BP_JVM_VERSION>
                        </env>
                    </image>
                </configuration>
            </plugin>

Development environment: Mac OS Sequoia Atlas CLI MongoDb 8 Java 23

Providers: MongoDb Atlas MongoDb 8 Google Cloud Platform / Cloud run Java 23

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 25 (12 by maintainers)

Most upvoted comments

I think you should engage Google support at this point. The goal being to understand why a server that appears to be ready and able to accept requests is unable to do so. Perhaps they can provide some diagnostics about the network.

If it’s not Mongo that’s the problem, then I would turn attention to the web server. Looking at the thread dump, you appear to be using Undertow (the XNIO threads). Have you tried with Tomcat or Jetty? If the problem doesn’t occur when using WebFlux (and assuming that you’re using the default of Netty there), that may be another pointer towards an Undertow problem.