testcontainers-java: Test container fails to stop when used as Spring bean

If a container is used as a bean in Spring application (sort of like an embedded container) that runs on Tomcat, the container fails to stop when shutdown signal is sent.

Container is configured like this:

@Bean(initMethod = "start")
public GenericContainer redisContainer() {
	return new GenericContainer("redis:3.2.9").withExposedPorts(6379);
}

@Bean
public JedisConnectionFactory redisConnectionFactory() {
	JedisConnectionFactory connectionFactory = new JedisConnectionFactory();
	connectionFactory.setHostName(redisContainer().getContainerIpAddress());
	connectionFactory.setPort(redisContainer().getMappedPort(6379));
	return connectionFactory;
}

Shutdown error:

14-Jun-2017 14:48:10.987 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2b9ff8a8]) and a value of type [org.testcontainers.shaded.io.netty.util.internal.InternalThreadLocalMap] (value [org.testcontainers.shaded.io.netty.util.internal.InternalThreadLocalMap@4581e709]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
14-Jun-2017 14:48:10.988 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2b9ff8a8]) and a value of type [org.testcontainers.shaded.io.netty.util.internal.InternalThreadLocalMap] (value [org.testcontainers.shaded.io.netty.util.internal.InternalThreadLocalMap@13da7c79]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
14-Jun-2017 14:48:10.988 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2b9ff8a8]) and a value of type [org.testcontainers.shaded.io.netty.util.internal.InternalThreadLocalMap] (value [org.testcontainers.shaded.io.netty.util.internal.InternalThreadLocalMap@34e88997]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This leaves Tomcat hanging.

The sample (a simple Spring Boot app) to reproduce the problem is available in this repo. To reproduce the problem either run the integrationTest Gradle task using ./gradlew integrationTest (this uses gretty to integration test the app on Tomcat 8), or build the WAR and run it on Tomcat (in this case the -Dspring.profiles.active=redisContainer VM parameter is needed).

To give some background - we’re in the process of migrating Spring Session integration tests to TestContainers (spring-projects/spring-session#798).

JUnit @ClassRule based integration tests were easy to migrate however the project also contains some sample application which are packaged as WARs and then integration tested on Tomcat 8 using gretty.

The idea was to provide a Spring profile which would start appropriate test container (in this case Redis) and configure the consumer (Redis connection factory) to connect to it but this has proved to be problematic due to reasons explained above.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 1
  • Comments: 36 (33 by maintainers)

Commits related to this issue

Most upvoted comments

@vpavic I got it working!

diff --git a/src/main/java/sample/TestContainerBeanApplication.java b/src/main/java/sample/TestContainerBeanApplication.java
index 01a912d..0df3997 100644
--- a/src/main/java/sample/TestContainerBeanApplication.java
+++ b/src/main/java/sample/TestContainerBeanApplication.java
@@ -42,7 +42,17 @@ public class TestContainerBeanApplication extends SpringBootServletInitializer
 
 		@Bean(initMethod = "start")
 		public GenericContainer redisContainer() {
-			return new GenericContainer("redis:3.2.9").withExposedPorts(6379);
+			return new GenericContainer("redis:3.2.9") {
+				@Override
+				public void close() {
+					super.close();
+					try {
+						dockerClient.close();
+					} catch (Exception e) {
+						e.printStackTrace();
+					}
+				}
+			}.withExposedPorts(6379);
 		}

Output:

14:42:19 INFO  Tomcat 8.0.33 started and listening on port 48080
14:42:19 INFO  testcontainer-bean runs at:
14:42:19 INFO    http://localhost:48080/testcontainer-bean
:integrationTest
Hello World!
:appAfterIntegrationTest
2017-07-07 14:42:19.678  INFO 4538 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2017-07-07 14:42:19.680  INFO 4538 --- [ost-startStop-1] o.a.c.c.C.[.[.[/testcontainer-bean]      : Closing Spring root WebApplicationContext
2017-07-07 14:42:19.681  INFO 4538 --- [ost-startStop-1] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@73ce805: startup date [Fri Jul 07 14:42:04 CEST 2017]; root of context hierarchy
2017-07-07 14:42:19.694  INFO 4538 --- [ost-startStop-1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
Server stopped.

BUILD SUCCESSFUL

Total time: 1 mins 37.544 secs

We need to close the client to allow JVM shutdown gracefully.

Here I use close method to call it because there was only 1 container, and this method must be called after all containers are closed.

Also as you can see it works with 1.3.0, no need to use the snapshot 🎉

@vpavic I’m working on it right now, but can’t promise anything at this moment 😃 So far it seems that the problem might come from docker-java and not TC itself. If so, I’ll try to workaround it, otherwise we have to wait until it’s fixed & released there

@dadoonet we’re waiting for a release of docker-java where this PR was merged: https://github.com/docker-java/docker-java/pull/980

once it’s released, we will be able to control Netty from our side and mark Netty threads as daemon threads, so that it will not prevent JVM from exiting.

I’ll update this issue once it’s done 😃

@vpavic oh, yes, we use Bintray to do the releases, so it’s in jcenter already, but will take ~15-20 min before appearing in Maven Central, sorry for not mentioning it 😃

Is there any current workaround for forcing those netty threads to close quickly? I would use System.exit(0) but that screws up maven exec:java. I’m successfully using TestContainers, Flyway and Jooq together to generate sources, but the 30 second delay on each build in a bit problematic.

@vpavic unfortunately such behaviour should not be shipped as a default one, because for most scenarios global client should stay even if container is stopped (because next test might start a new container)

My workaround should work with 1.3.0/1.3.1 and very specific to your use case. We might think how to support such use case later, 1.5.0 maybe, but for now I recommend you to do it like that 😃

Just to give you a bit more of technical background of the issue:

  1. Testcontainers starts a client
  2. starts a container
  3. Spring stops it on context’s close
  4. client remains open, someone should close it as well, but not before the container is stopped

What workaround does: When Spring calls .stop() on that container, we first call container’s stop logic, and then close the global client.

Yeah, I saw your comment so I gave it a quite a bit more than 30 sec but it didn’t help. OK, will try to build it myself and let you know.