testcontainers-java: GenericContainer#withCopyFileToContainer is not efficient memory-wise

Hi,

We’ve recently seen OOM in our tests and after debugging by generating a heap dump and opening it, we found that for the following code that we use in XWiki tests, TC is copying all the content of the sourceDirectory in memory:

MountableFile mountableDirectory = MountableFile.forHostPath(sourceDirectory);
container.withCopyFileToContainer(mountableDirectory, targetDirectory);

Since our directory contains 300MB+ of files, the memory we give to the JVM is not enough.

Note that we copy and don’t bind because we need to have it work in the DOOD use case. Our code is:

    protected void mountFromHostToContainer(GenericContainer<?> container, String sourceDirectory,
        String targetDirectory)
    {
        // Note 1: File mounting is awfully slow on Mac OSX. For example starting Tomcat with XWiki mounted takes
        // 45s+, while doing a COPY first and then starting Tomcat takes 8s (+5s for the copy).
        // Note 2: For the DOOD use case, we also do the copy instead of the volume mounting since that would require
        // to have the sourceDirectory path mounted from the host and this would put and leave files on the host which
        // would not work with parallel executions (think about multiple CI jobs executing in parallel on the same host)
        // and would also not be clean.
        String osName = System.getProperty("os.name").toLowerCase();
        if (isInAContainer() || osName.startsWith("mac os x")) {
            MountableFile mountableDirectory = MountableFile.forHostPath(sourceDirectory);
            container.withCopyFileToContainer(mountableDirectory, targetDirectory);
        } else {
            container.withFileSystemBind(sourceDirectory, targetDirectory);
        }
    }

It would be great if TC could stream the data instead of keeping it in memory. Is that possible?

Thanks

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 17 (10 by maintainers)

Commits related to this issue

Most upvoted comments

Is it possible to give some priority to this? I believe putting the docker compose file in the root of the project is something a lot of people do (Even start.spring.io generates it like that). All the duplicates of this issue (#7239, #1348, #2863) are a testament to this being quite an issue for a lot of people currently.

Even some mechanism to tell testcontainers to not copy every file, but just .env for example would already be great I guess.

@kiview Hello! Any chance of having the patch at https://github.com/alexanderankin/testcontainers-java/commit/21cbc05d6633bcff443478c7844851fbf5e1ea66 be included in the next release of TC by any chance? 😃 Thx

Yes, I tried a lot of workaround but I’m stuck. I keep version 1.8.0 😕 no choice

Thanks for your investigation @vmassol. You are right and we will look into improving this.