testcontainers-java: [Bug]: Shutdown hook in RyukResourceReaper prevents a graceful shutdown with spring framework

Module

Core

Testcontainers version

1.19.7

Using the latest Testcontainers version?

Yes

Host OS

Linux, Windows

Host Arch

x86_64

Docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:56 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

What happened?

Dears,

With the introduction of this feature: #7717

The ryuk instance is terminated by a shutdown hook, therefore all containers are closed very early.

This prevents other shutdown hooks (eg: those set by the spring framework) to perform actions needed for a graceful shutdown that still require containers to be up. Due to a retry mechanism in Spring’s shutdown hook, it greatly increases the overall time to run tests!

Could you please revert this change or introduce a mean to disable this shutdown hook, thanks!

Relevant log output

No response

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 13
  • Comments: 19 (6 by maintainers)

Commits related to this issue

Most upvoted comments

Hey @jobayle,

Due to a retry mechanism in Spring’s shutdown hook, it greatly increases the overall time to run tests!

This is of course a situation we would like to solve. What absolute amount of time are we talking abut here?

I am not sure https://github.com/testcontainers/testcontainers-java/pull/8732 is the right approach though, since this would still require all affected users, to apply this config option (not ideal). We could make configuration more programmatically accessible, but this would also require downstream integration changes by the Spring project.

Since a considerable amount of Testcontainers users are also Spring users, I think reverting the change and accept the containers lingering for some time after process termination can be the better DX, WDYT @eddumelendez?

The registering of shutdown hook also has other problems, such as classloader issues in some specific use-cases. Considering this, I vote for reverting the change that introduced the shutdown hook. Or at least made it configurable (and ‘off’ by default).

The solution of this problem will also solve the problem of using Testcontainers from Maven (e.g. #7923, #1454, testcontainers/testcontainers-jooq-codegen-maven-plugin#37) if shutdown hook will not be registered by default or at least configurable.

Hi @kiview, sure: Here’s the first and last log of the SpringApplicationShutdownHook:

16:15:25.372 [SpringApplicationShutdownHook] INFO - UPDATING MACHINE CONFIGURATION STATUS=SHUTDOWN
[...] lots of connection errors with DB/redis/...
16:16:27.222 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-13 - Shutdown completed.

So roughly a whole minute!

By enabling container reuse to simulate how things worked before #7717, the shutdown is almost instant:

16:38:34.259 [SpringApplicationShutdownHook] INFO  - UPDATING MACHINE CONFIGURATION STATUS=SHUTDOWN
16:38:34.749 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-3 - Shutdown initiated...
16:38:34.751 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-3 - Shutdown completed.
16:38:34.753 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Shutdown initiated...
16:38:34.755 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-2 - Shutdown completed.
16:38:34.755 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown initiated...
16:38:34.758 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Shutdown completed.
16:38:34.806 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-9 - Shutdown initiated...
16:38:34.807 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-9 - Shutdown completed.
16:38:34.810 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-8 - Shutdown initiated...
16:38:34.813 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-8 - Shutdown completed.
16:38:34.813 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-7 - Shutdown initiated...
16:38:34.816 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-7 - Shutdown completed.
16:38:34.856 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-15 - Shutdown initiated...
16:38:34.857 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-15 - Shutdown completed.
16:38:34.860 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-14 - Shutdown initiated...
16:38:34.862 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-14 - Shutdown completed.
16:38:34.862 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-13 - Shutdown initiated...
16:38:34.864 [SpringApplicationShutdownHook] INFO  com.zaxxer.hikari.HikariDataSource - HikariPool-13 - Shutdown completed.

Yes, I’d really love a fix or a revert, thanks 🙂

Another update: I ran ./gradlew test --no-daemon and it ran in 1m 57s.

My focus has been on test execution time, dependencies were already downloaded.

Just sharing some updates. Thanks to the reproducer I can see how the gradle project takes 2m 40s on my machine and the shutdown takes a lot of time. For testing purposes I did the same migrating the project to maven and it ran under 1m 20s. I didn’t catch it due to the projects I shared before were only maven projects. It’s interesting!

This is just an update and not a statement about not fixing it or saying move to maven. I’ll be deep dive a little bit more but probably the easiest and quick solution would be reverting the change as it has been suggested.

Thanks again for the reproducer, it helps to move things faster!

Hi,

I’ve spent a few hours investigating the issue and making a reproducer: https://github.com/jobayle/repro-tc-shutdownhook

See the shutdown time when the integration tests are run in a github action : https://github.com/jobayle/repro-tc-shutdownhook/commit/fa7dc3b95b7450808ad63f1223c3baf6a8dc9717/checks/32354065819/logs

first line:

2024-10-31T18:40:07.050Z INFO 2089 — [ionShutdownHook] bug.machine.ApplicationStatus : UPDATING MACHINE CONFIGURATION STATUS=SHUTDOWN

last line:

2024-10-31T18:41:12.293Z INFO 2089 — [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-5 - Shutdown completed.

More than one minute.

Thanks @EAlf91, but imho, this is not a viable solution if we have to maintain a patched version of testcontainers.

@jobayle It’s mentioned here: https://github.com/testcontainers/testcontainers-java/issues/7871#issuecomment-2041169911 but it would require a change I guess. The author made the hook configurable via spring properties: https://github.com/alex-arana/testcontainers-java/commit/a0fcbdf001ee17b5649882052ec62ad7092d1a74

I didn’t try it myself and didn’t see any way to configure this without using this fork. I’d highly appreciate to see a revert of #7717 or maybe using this way to configure it via spring properties