testcontainers-java: [Bug]: Container Shutdown behavior changed between 1.19.1 and 1.19.2 ressources not cleaned up properly

Module

Core

Testcontainers version

1.19.2

Using the latest Testcontainers version?

Yes

Host OS

Linux

Host Arch

x86

Docker version

Docker version 24.0.5, build ced0996

What happened?

We’re facing problems running our integration tests using Testcontainers since 1.19.2 and even though our tests run fine, we’re facing a lot of errors and flooded log outputs.

out test setup looks like this:

@SpringBootTest
@ActiveProfiles("test")
public class BaseIntegrationTest {

    @Autowired
    EntityManager entityManager;


    @Autowired
    JdbcTemplate jdbcTemplate;

    protected static final String ACCOUNT_ID = "000000000000";
    private static final String CONTAINER_PATH = "/etc/localstack/init/ready.d/";
    public static final LocalStackContainer LOCALSTACK = new LocalStackContainer(DockerImageName.parse("localstack/localstack"))
            .withCopyFileToContainer(MountableFile.forHostPath("localstack/init.sh"), CONTAINER_PATH)
            .waitingFor(Wait.forLogMessage(".*Setup done, ready to process*\\n", 1))
            .withServices(SQS, SNS, S3)
            .withEnv("AWS_DEFAULT_REGION", Region.EU_CENTRAL_1.id())
            .withEnv("ENVIRONMENT", "test");

    static {
        LOCALSTACK.start();

        System.setProperty("spring.cloud.aws.endpoint", LOCALSTACK.getEndpoint().toString());
        System.setProperty("spring.cloud.aws.credentials.access-key", LOCALSTACK.getAccessKey());
        System.setProperty("spring.cloud.aws.credentials.secret-key", LOCALSTACK.getSecretKey());
        System.setProperty("ACCOUNT_ID", ACCOUNT_ID);
        System.setProperty("ENVIRONMENT", "test");
    }

    @BeforeEach
    public void init() {
        truncate();
    }

    protected void truncate() {
        jdbcTemplate.execute("some table");
    }

when completing our tests we face a lot of errors due to our sqs listeners not being able to poll messages anymore.

The complete log output can be found in the relevant log output section

2023-11-24T08:16:05.965+01:00 ERROR 16588 --- [nc-response-1-6] i.a.c.s.l.s.AbstractPollingMessageSource : Error polling for messages in queue https://sqs.eu-central-1.amazonaws.com/000000000000/some_test_queue

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: The connection was closed during the request. The request will usually succeed on a retry, but if it
 does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely ma
nner. Channel Information: ChannelDiagnostics(channel=[id: 0x7eadd6af, L:/127.0.0.1:57073 ! R:/127.0.0.1:56971], channelAge=PT0.0129724S, requestCount=1)
        at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)

Relevant log output

2023-11-24T08:16:05.965+01:00 ERROR 16588 --- [nc-response-1-6] i.a.c.s.l.s.AbstractPollingMessageSource : Error polling for messages in queue https://sqs.eu-central-1.amazonaws.com/000000000000/some_test_queue

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: The connection was closed during the request. The request will usually succeed on a retry, but if it
 does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely ma
nner. Channel Information: ChannelDiagnostics(channel=[id: 0x7eadd6af, L:/127.0.0.1:57073 ! R:/127.0.0.1:56971], channelAge=PT0.0129724S, requestCount=1)
        at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:165)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider disabling a
ny proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely manner. Channel Information: Chan
nelDiagnostics(channel=[id: 0x7eadd6af, L:/127.0.0.1:57073 ! R:/127.0.0.1:56971], channelAge=PT0.0129724S, requestCount=1)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
        ... 22 common frames omitted
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: The connection was closed during the request. The request will usually succeed on a retry, but 
if it does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a time
ly manner. Channel Information: ChannelDiagnostics(channel=[id: 0x82f0c219, L:/127.0.0.1:57027 ! R:/127.0.0.1:56971], channelAge=PT1.3915045S, requestCount=1)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Eine bestehende Verbindung wurde softwaregesteuert
durch den Hostcomputer abgebrochen
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: The connection was closed during the request. The request will usually succeed on a retry, but 
if it does not: consider disabling any proxies you have configured, enabling debug logging, or performing a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a time
ly manner. Channel Information: ChannelDiagnostics(channel=[id: 0xf9454d2c, L:/127.0.0.1:57061 ! R:/127.0.0.1:56971], channelAge=PT0.0134961S, requestCount=1)
Caused by: java.io.IOException: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider disabling any proxies you have configured, enabling debug logging, or performing
 a TCP dump to identify the root cause. If this is a streaming operation, validate that data is being read or written in a timely manner. Channel Information: ChannelDiagnostics(channel=[id: 0x7eadd6af, L:/127.0.0.1:57073 ! R:/127.0
.0.1:56971], channelAge=PT0.0129724S, requestCount=1)
        at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:472)
        at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelInactive(ResponseHandler.java:222)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:206)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
        at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:411)
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:376)
        at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:328)
        at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
        at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
        at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
        at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        ... 1 common frames omitted

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 17 (4 by maintainers)

Most upvoted comments

Hello, guys, is there any progress or news regarding this topic?

Can this be reopened? The change from #7717 actually broke shutdown procedure in a lot of Spring Boot applications and there’s no way to fix it without reverting back to 1.19.1.

In this thread it was incorrectly stated that Ryuk will shut down immediately after JVM shutdown, but the truth is that Ryuk now receives SIGTERM the moment JVM receives SIGTERM which introduces race condition between Spring’s shutdown hook and Ryuk.

I can confirm that the issue occurs when I bump the version fom 1.19.1 to 1.19.2 where a shutdown hook has been added.

@eddumelendez - Could the shutdown hook registration be configurable so devs could decide if they want to use new or legacy approach which works Spring context?

I agree with @xak2000, test containers are widely used in Spring applications tests so I think that integration should also be tested before releasing a new version of test containers. Spring major version has been not bumped up so it means that testcontainers should work with all 6.x.x versions.

DirtyContext is not an option for apps with a lot of tests. I want to use shared context between tests to speed them up.

@eddumelendez Is it possible to add an ability to optionally disable the registering of this shudown hook? If I understand you correctly, ryuk will be runned for 10s after JVM is shut down. Is it guaranteed? If so, then it will allow all Spring Application Contexts to properly shut down during these 10 seconds.

At the moment after upgrading to Spring Boot 3.2.2 (and TestContainers 1.19.3) this behavior just makes any logic, that executes during Spring Application shutdown phase, and requires a container to exist, to fail. One example I described here: https://github.com/testcontainers/testcontainers-java/discussions/7454#discussioncomment-8310094

Or what else can be done except marking each method with @DirtiesContext to guarantee the order of shutting down the Spring Context and TestContainers? Maybe TestContainers devs can work with Spring devs to find a solution? 😃

I’ll add my 5 cents to the last comment.

The problem with this “just logging issue” is that it is not always just logging issue, but sometimes it is also actual behavior issue.

As I alredy described here there are legitimate cases when Spring Framework (or one of its libraries) tries to execute an SQL query when Spring Context is closing. One example is DefaultLockRepository.close() from the Spring Integration project, that is not even a 3rd party library, but is an official 1st party library for Spring Framework. The method is designed to be called when Spring Application Context is closing. So, the Application in tests behaves not as expected. Tests still pass, yes. But many exceptions are thrown during shutdown. Also SQL connection pool (HikariCP in my case) can’t understand what is going on and why all its connections in the pool suddenly became broken, so it waits (30 seconds by default) before giving up. This increases tests execution time by [30 second] * [number of started contexts]. And a lot of garbage log messages with stacktraces because all connections in the pool are dead and no new connections are possible to create.

It’s just my experience. Who knows what else libraries this behavior could affect and in what way. I can’t believe that this behavior is considered as OK now. What I believe is that it is probably not that easy to synchronize the shutdown of TestContainers with the shutdown of Spring Application Context. But this doesn’t mean this problem should be ignored. Let’s try to find a solution together! And while it is not found yet, at least return the old behavior of 10s delay before shutdown TestContainers. Probably with a config property to override this behavior, if these 10s more time of living containers is really a problem for someone.

You could argue that an application should be written in a way that non-functional external services should not prevent it from proper shutdown (12-factors and all this stuff) and I would agree, but this should not be “normal” situation that reproduces on every test run. A separate test should test this scenario, not all the tests that just test normal app behaviour scenarios.

You can see others reporting the same in previous versions.