testcontainers-java: [Bug]: Shutdown hook in RyukResourceReaper prevents a graceful shutdown with spring framework
Module
Core
Testcontainers version
1.19.7
Using the latest Testcontainers version?
Yes
Host OS
Linux, Windows
Host Arch
x86_64
Docker version
Client:
Cloud integration: v1.0.35+desktop.13
Version: 26.0.0
API version: 1.45
Go version: go1.21.8
Git commit: 2ae903e
Built: Wed Mar 20 15:18:56 2024
OS/Arch: windows/amd64
Context: default
Server: Docker Desktop 4.29.0 (145265)
Engine:
Version: 26.0.0
API version: 1.45 (minimum version 1.24)
Go version: go1.21.8
Git commit: 8b79278
Built: Wed Mar 20 15:18:01 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.28
GitCommit: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
What happened?
Dears,
With the introduction of this feature: #7717
The ryuk instance is terminated by a shutdown hook, therefore all containers are closed very early.
This prevents other shutdown hooks (eg: those set by the spring framework) to perform actions needed for a graceful shutdown that still require containers to be up. Due to a retry mechanism in Spring’s shutdown hook, it greatly increases the overall time to run tests!
Could you please revert this change or introduce a mean to disable this shutdown hook, thanks!
Relevant log output
No response
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 13
- Comments: 19 (6 by maintainers)
Commits related to this issue
- GH-8558 Make Ryuk shutdown hook configurable. — committed to mmaeller/testcontainers-java by mmaeller 2 years ago
- GH-8558 Format. — committed to mmaeller/testcontainers-java by mmaeller 2 years ago
- Revert "Add shutdownHook to send sigterm to ryuk (#7717)" This reverts commit 39f02190c01702fb3121b080f29c72bbdf7c78ea. Fixes #8558 — committed to testcontainers/testcontainers-java by eddumelendez 2 years ago
- Revert "Add shutdownHook to send sigterm to ryuk (#7717)" (#9504) This reverts commit 39f02190c01702fb3121b080f29c72bbdf7c78ea. Fixes #8558 — committed to lonecalvary78/testcontainers-java by eddumelendez 2 years ago
Hey @jobayle,
This is of course a situation we would like to solve. What absolute amount of time are we talking abut here?
I am not sure https://github.com/testcontainers/testcontainers-java/pull/8732 is the right approach though, since this would still require all affected users, to apply this config option (not ideal). We could make configuration more programmatically accessible, but this would also require downstream integration changes by the Spring project.
Since a considerable amount of Testcontainers users are also Spring users, I think reverting the change and accept the containers lingering for some time after process termination can be the better DX, WDYT @eddumelendez?
The registering of shutdown hook also has other problems, such as classloader issues in some specific use-cases. Considering this, I vote for reverting the change that introduced the shutdown hook. Or at least made it configurable (and ‘off’ by default).
The solution of this problem will also solve the problem of using Testcontainers from Maven (e.g. #7923, #1454, testcontainers/testcontainers-jooq-codegen-maven-plugin#37) if shutdown hook will not be registered by default or at least configurable.
Hi @kiview, sure: Here’s the first and last log of the
SpringApplicationShutdownHook:So roughly a whole minute!
By enabling container reuse to simulate how things worked before #7717, the shutdown is almost instant:
Yes, I’d really love a fix or a revert, thanks 🙂
Another update: I ran
./gradlew test --no-daemonand it ran in 1m 57s.My focus has been on test execution time, dependencies were already downloaded.
Just sharing some updates. Thanks to the reproducer I can see how the gradle project takes 2m 40s on my machine and the shutdown takes a lot of time. For testing purposes I did the same migrating the project to maven and it ran under 1m 20s. I didn’t catch it due to the projects I shared before were only maven projects. It’s interesting!
This is just an update and not a statement about not fixing it or saying move to maven. I’ll be deep dive a little bit more but probably the easiest and quick solution would be reverting the change as it has been suggested.
Thanks again for the reproducer, it helps to move things faster!
Hi,
I’ve spent a few hours investigating the issue and making a reproducer: https://github.com/jobayle/repro-tc-shutdownhook
See the shutdown time when the integration tests are run in a github action : https://github.com/jobayle/repro-tc-shutdownhook/commit/fa7dc3b95b7450808ad63f1223c3baf6a8dc9717/checks/32354065819/logs
first line:
last line:
More than one minute.
Thanks @EAlf91, but imho, this is not a viable solution if we have to maintain a patched version of testcontainers.
@jobayle It’s mentioned here: https://github.com/testcontainers/testcontainers-java/issues/7871#issuecomment-2041169911 but it would require a change I guess. The author made the hook configurable via spring properties: https://github.com/alex-arana/testcontainers-java/commit/a0fcbdf001ee17b5649882052ec62ad7092d1a74
I didn’t try it myself and didn’t see any way to configure this without using this fork. I’d highly appreciate to see a revert of #7717 or maybe using this way to configure it via spring properties