dokka: dokkaJavadocJar fires "java.lang.OutOfMemoryError: Metaspace"

OSS maintainers often feel it’s a thankless and underappreciated job, but any paid project with an OOM error for 6 months would be shut down and the project members would be looking for new jobs. But not if it’s OSS 😃

Long story short: it’s a memory leak (mostly) caused by kotlinx.coroutines https://github.com/Kotlin/kotlinx.coroutines/issues/2558

Dokka 1.4.30 will (partially) fix this. There’s actually nothing more we can do apart from force-closing dispatchers right now. We could remove coroutines from our code (and/or use JVM’s threads), but this would be a painful, long job with the effect of degraded performance (initially Dokka was single threaded and slower by several orders of magnitude).

Giving the JVM more heap and metaspace memory and not reusing the Gradle daemon (which holds all the references that prevent the GC from doing its job) is still a viable workaround that we personally use

We’ve fixed all known Metaspace leaks as of today, and added some tests to detect such problems in the future. Unfortunately, Dokka still requires a considerable amount of memory to document large projects.

If you are experiencing Metaspace-related issues when running Dokka:

• Upgrade to the latest version of Dokka
• Try increasing Metaspace size (-XX:MaxMetaspaceSize=2G) as the defaults can be modest. The larger the project, the more you might need to allocate.

If you you have suspicions that Dokka is leaking memory or the Metaspace size you’ve set goes beyound reasonable, please create a separate issue for your case. We would appreciate having a reproducer and as much information as possible to collect.

I’m closing this general issue so that we can concentrate on specific cases.

Thanks - re: https://github.com/Kotlin/dokka/issues/1405#issuecomment-1021795270, I think my problem might have been that I was using a very old JDK to build (JDK 8). I upgraded to a newer one and was not able to reproduce the problem any more. If it’s not that, I’m not sure what it was… I agree it doesn’t make sense that heap was filling up and the MaxMetaspaceSize flag seemed to be mitigating it, since those are two different things. Sorry for distracting from the main issue - just thought I should follow up here for reference in case anyone else hits the same problem.

Thank you so much for the updates @kamildoleglo

@asarkar open source software benefits everyone, it would be great if more people paid to help support it (or better yet, understand it and contribute to it) but that sadly is not the reality we live in. You are always welcome to find and propose solutions though!

Note for the Metaspace leak fix in 1.6.10

Dokka loads a bunch of classes (especially for large multi-module projects), so it relies heavily on Metaspace. Even if there’s no leak, you may still need to increase -XX:MaxMetaspaceSize to the point where dokka is able to load all of the classes it needs to generate documentation. I’m not exactly sure on sizing, but anything up to 512M/1G should be plenty for the majority of projects. If it requires more than that, it may indicate a leak or a very big project/module, worth reporting/checking.

The difference between 1.6.0 and 1.6.10 is that for versions <= 1.6.0 classes weren’t unloaded and metaspace didn’t get garbage collected. So Metaspace was filling up and never freed, causing java.lang.OutOfMemoryError: Metaspace. The fix was to give it so much Metaspace memory that it would finish the build before running out of it.

For >= 1.6.10, classes are unloaded (after a module run, for instance) and metaspace gets collected regularly, so dokka needs less -XX:MaxMetaspaceSize to finish the build. If you had to set -XX:MaxMetaspaceSize to anything larger than 1GB, you may try decreasing it until it fails (maybe you’ll even be able to remove it altogether, depends on the project).

1.6.10 is mostly an enabler for larger projects that coduln’t be built previously (or required too much metaspace), like aws-sdk-kotlin with 260 generated modules. On 1.6.0 with -XX:MaxMetaspaceSize=1G it fails after 10 minutes of running. On 1.6.10 with -XX:MaxMetaspaceSize=500M it completes in 2.5 hours.

update: Unfortunately, it seems like the metaspace leak is fixed only if you don’t have the samples configuration property set. If you do have it set and experience the leak, try removing it, should help. We’ll look into solving this as well.

tasks.withType<DokkaTask>().configureEach {
    dokkaSourceSets {
        configureEach {
            samples.from("samples/basic.kt", "samples/advanced.kt")
    }
}

This should be fixable now that Coroutines 1.6.0 allows you to shutdown the dispatchers.

Can confirm the issue remains with Dokka 1.4.30 Luckily org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m works.

Recently, we’ve set it to 2G to make it work 😭

Yup, https://github.com/Kotlin/dokka/commit/b298da4ce1a55a6f69e7ed00020ed3af558f3750 should be in

This issue is even more problematic since the Gradle Daemon doesn’t die but rather hangs so you have to manually kill it. The error in question something like this:

  Process exited with code 1
    Gradle failure report
    FAILURE: Build failed with an exception.

    * What went wrong:
    Gradle could not start your build.
    > Could not create service of type ResourceSnapshotterCacheService using GradleUserHomeServices.createResourceSnapshotterCacheService().
       > Timeout waiting to lock file hash cache (/Users/user/.gradle/caches/6.6/fileHashes). It is currently in use by another Gradle instance.
         Owner PID: 71496
         Our PID: 71896
         Owner Operation:
         Our operation:
         Lock file: /Users/user/.gradle/caches/6.6/fileHashes/fileHashes.lock

I’m having the exact same experience in a rather small multi-module project. Already tried to increase metaspace and to disable parallelism but it didn’t provide any deterministic results. Any updates on this? 🙏

This issue is even more problematic since the Gradle Daemon doesn’t die but rather hangs so you have to manually kill it. The error in question something like this:

  Process exited with code 1
    Gradle failure report
    FAILURE: Build failed with an exception.

    * What went wrong:
    Gradle could not start your build.
    > Could not create service of type ResourceSnapshotterCacheService using GradleUserHomeServices.createResourceSnapshotterCacheService().
       > Timeout waiting to lock file hash cache (/Users/user/.gradle/caches/6.6/fileHashes). It is currently in use by another Gradle instance.
         Owner PID: 71496
         Our PID: 71896
         Owner Operation:
         Our operation:
         Lock file: /Users/user/.gradle/caches/6.6/fileHashes/fileHashes.lock

In my case I had to bump the metaspace even further (1024m). I have a multi-module project with 8 modules, half of which are multiplatform. Note that I’m running dokkaHtml, not dokkaJavadoc, but I get the same error with javadoc.

Also thanks for suggesting a workaround. I’ll repost it for the better visibility.

Workaround:

put org.gradle.jvmargs=-XX:MaxMetaspaceSize=512m in gradle.properties

I am working on a replacement for the Dokka Gradle Plugin, https://github.com/adamko-dev/dokkatoo, which uses the Gradle Worker API to run Dokka in an isolated process.

I had hoped that running Dokka in an isolated daemon would help, especially as Gradle should re-use the daemon, but unfortunately it hasn’t helped, and anecdotally OOM errors seem to be triggered more often for even the basic examples andintegration-test projects.

Here is an example failure: https://github.com/adamko-dev/dokkatoo/actions/runs/4211535460/jobs/7310041760

Since I have set up Dokka to run in a separate process it should be possible to analyze the process independently of other work. I will see if I can set this up, but if anyone would like to try, please check out the project and run the tests, and hints or tips would be appreciated!

@GlenKPeterson is gradle.properties.kts a thing? Shouldn’t it be just plain gradle.properties?

## gradle.properties
org.gradle.jvmargs=-XX:MaxMetaspaceSize=2G

If you’re not using samples, you should not experience any major leaks. If it’s an open source project, I can have a look, but if it’s closed source, there’s little I can do.

Try increasing metaspace and have a look at the Metaspace Graph in VisualVM. You should see the saw-like pattern just like in the screenshot above. If it only goes up and never down, try pressing Perform GC button, it should stall for a few seconds and then drop down. If it doesn’t happen, there might be a leak. Generally any info that you can gather would be useful, we’d like to get this fixed if it’s a leak.

If you need help or have questions, feel free to reach out to me at ignat.beresnev@jetbrains.com or by messaging me in the Kotlin Community Slack (same name there)

Thanks for your response. I’m always using --no-daemon at the command line. I think having IntelliJ open causes a separate daemon to run?

The Gradle Tooling API that is used by IDEs and other tools to integrate with Gradle always uses the Gradle Daemon to execute builds. If you are executing Gradle builds from within your IDE you are using the Gradle Daemon and do not need to enable it for your environment. https://docs.gradle.org/current/userguide/gradle_daemon.html#sec:tools_and_ides

Could that be what’s messing me up?

I searched my entire project for “samples” and only found an image: sampleScreenShot.png. I’m still out of metaspace on and off.

I only have one gradle.properties.kts in the master project with org.gradle.jvmargs="-XX:MaxMetaspaceSize=1024m". I assume that’s the right place for it.

In our project the problem disappeared after upgrading to Dokka 1.6.10. I guess this ticket can be closed now.

The many-partial approach.

Before running the final ./gradlew docs command, we run a bunch of ./gradlew docsPartial commands. Each command spawns a one-time gradle daemon, but is small enough to run without memory problems in this daemon.

In the work-around we run this as:

./gradlew --stop
./gradlew docsPartial -PdocsPartialPrefix=api_ --no-daemon
./gradlew docsPartial -PdocsPartialPrefix=core_ --no-daemon
./gradlew docsPartial -PdocsPartialPrefix=media_ --no-daemon
./gradlew docsPartial -PdocsPartialPrefix=stock_ --no-daemon
./gradlew docs --no-daemon

The docsPartial task is added to the build.gradle.kts file as follows:

tasks.register("docsPartial") {
    description = "Generate intermediate documentation for modules who's name starts with" +
        " property 'docsPartialPrefix'"
    group = "Documentation"

    (findProperty("docsPartialPrefix") as String?)?.let { prefix ->
        println("docsPartialPrefix: $prefix")
        subprojects.filter { it.name.startsWith(prefix) }.forEach { module ->
            module.tasks.findByName("dokkaHtmlPartial")?.let { task ->
                dependsOn(":${module.name}:dokkaHtmlPartial")
            }
        }
    } ?: doFirst {
        throw GradleException(
            "You must set property 'docsPartialPrefix' when calling this task."
        )
    }
}

As this issue is open for 16 months now, I assume it is a tough one to solve. Therefore we took the pragmatic approach and tried to work around the problem. The development project that I am working on is growing and these out-of-memory issues with Dokka are getting more and more annoying.

With the information of other people that have investigated the issue, see previous posts on this ticker, we found two ways to work around this problem.

• Kill the gradle daemon when it uses an excessive amount of memory; then restart the same gradle dokka task. The ./gradlew --status command shows status STOPREQUESTED for a gradle daemon that is running short on memory. However, the daemon only stops when the current gradle tasks are done. So the gradle dokka command keeps using the same daemon all the time, but finally fails due to memory overflow in this daemon. Forced killing of such a daemon will fail the gradle task. When starting the same gradle command again, a fresh daemon is started. This new gradle command continues where the previous one left off, as there is no need to redo the dokka-partial tasks that have already succeeded.
• Run the dokkaHtmlPartial tasks individually or clustered before running the dokkaHtmlMultiModule task, in separate invocations of the gradle command. At the end of each gradle invocation, the daemon will be stopped when it uses too much memory. Even better is to run gradle with the ‘–no-daemon’ option, which starts a one-time daemon just for this gradle invocation.

The daemon-kill approach is quite rude, but it scales nicely because Gradle detects when the daemon uses an excessive amount of memory. The many-partial approach is more structured, but finding the best clustering of dokka-partial tasks can be a tedious job, which may need to be redone when the project grows or gets reorganised. Also this approach takes more execution time. Gradle sometimes cannot always use the optimal amount of worker threads, but also each gradle invocation will redo the gradle configuration phase.

For more details of each work-around see my next posts on this ticket.

Hi, I have just tried upgrade from Gradle 6.9 to Gradle 7.1.1 but our Dokka 0.10.1 plugin is failing with:

* What went wrong:
Some problems were found with the configuration of task ':dokka' (type 'DokkaTask').
  - In plugin 'org.jetbrains.dokka' type 'org.jetbrains.dokka.gradle.DokkaTask' property 'configuration.collectKotlinTasks' is missing an input or output annotation.
    
    Reason: A property without annotation isn't considered during up-to-date checking.
    
    Possible solutions:
      1. Add an input or output annotation.
      2. Mark it as @Internal.
    
    Please refer to https://docs.gradle.org/7.1.1/userguide/validation_problems.html#missing_annotation for more details about this problem.
  - In plugin 'org.jetbrains.dokka' type 'org.jetbrains.dokka.gradle.DokkaTask' property 'configuration.name' is missing an input or output annotation.
    
    Reason: A property without annotation isn't considered during up-to-date checking.
    
    Possible solutions:
      1. Add an input or output annotation.
      2. Mark it as @Internal.
    
    Please refer to https://docs.gradle.org/7.1.1/userguide/validation_problems.html#missing_annotation for more details about this problem.
  - In plugin 'org.jetbrains.dokka' type 'org.jetbrains.dokka.gradle.DokkaTask' property 'dokkaRuntime' is missing an input or output annotation.
    
    Reason: A property without annotation isn't considered during up-to-date checking.
    
    Possible solutions:
      1. Add an input or output annotation.
      2. Mark it as @Internal.

Unfortunately, upgrading to Dokka 1.4.30 breaks due to OOM, and even 2GB of Metaspace isn’t enough to avoid that 🤯.

Seeing as the latest Dokka plugin has an apparently unfixable memory leak, can someone release a 0.10.2 version that is compatible with Gradle 7.1 please?

I’m still seeing the issue when simply running ./gradlew clean docs, but --no-daemon seems to be the only flag required to get it to complete now! Thanks for all your work @kamildoleglo

Nope, I’ve forgot to do the grooming, sorry