anvil: Compiler doesn't pick incremental changes to multibindings in some circumstances

Does anyone found a workaround that allows updating Kotlin version without disabling incremental compilation?

Yes.

TL;DR fix

To solve it for yourself immediately, put this in your Anvil project(s) or in a convention plugin:

tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompile>().configureEach {
  // Kapt tasks are also KotlinCompile tasks, but we don't care about them
  if (this !is org.jetbrains.kotlin.gradle.tasks.KaptGenerateStubs) {

    val anvilSrcGenDir = layout.buildDirectory.dir(sourceSetName.map{ "anvil/src-gen-$it/anvil" })

    // adds the Anvil directory to the task's outputs
    this.outputs.dir(anvilSrcGenDir)
  }
}

tasks.withType(org.jetbrains.kotlin.gradle.tasks.KotlinCompile).configureEach {
  // Kapt tasks are also KotlinCompile tasks, but we don't care about them
  if (!(this instanceof org.jetbrains.kotlin.gradle.tasks.KaptGenerateStubs)) {

    def anvilSrcGenDir = layout.buildDirectory.dir(sourceSetName.map { "anvil/src-gen-$it/anvil" })

    // adds the Anvil directory to the task's outputs
    this.outputs.dir(anvilSrcGenDir)
  }
}

Why?

Anvil’s output directory isn’t being added to the KotlinCompile task’s outputs. This is why incremental compilation (and compilation from a remote build cache) is so hit-or-miss.

Assume you have a single :lib kotlin-jvm library, and this is the only source:

package com.foo

import com.squareup.anvil.annotations.ContributesMultibinding

@ContributesMultibinding(String::class)
class MyClass : MyInterface

interface MyInterface

Handling Deletions and Build Cache

This script reproduces running a build on an Anvil module where the sources aren’t in the build directory, but they are cached (locally, in this case).

./gradlew compileKotlin
rm -rf lib/build
./gradlew compileKotlin -i

The second compileKotlin task will be FROM-CACHE, because Gradle does know that it needs to run something. The console output will be:

> Task :lib:compileKotlin FROM-CACHE
[...]
Build cache key for task ':lib:compileKotlin' is 8f3471179429d249f958ea6ad1380a62
Task ':lib:compileKotlin' is not up-to-date because:
  Output property 'classpathSnapshotProperties.classpathSnapshotDir' file [...]/lib/build/kotlin/compileKotlin/classpath-snapshot has been removed.
  Output property 'classpathSnapshotProperties.classpathSnapshotDir' file [...]/lib/build/kotlin/compileKotlin/classpath-snapshot/shrunk-classpath-snapshot.bin has been removed.
  Output property 'destinationDirectory' file [...]/lib/build/classes/kotlin/main has been removed.
Loaded cache entry for task ':lib:compileKotlin' with cache key 8f3471179429d249f958ea6ad1380a62

Note that there are only three outputs listed, and none of them have anything to do with Anvil:

build/kotlin/compileKotlin/classpath-snapshot
build/kotlin/compileKotlin/classpath-snapshot/shrunk-classpath-snapshot.bin
build/classes/kotlin/main

Those three outputs were restored from cache, but the Anvil-generated code was not. That means no hints, no factories, and somewhere downstream, no Dagger bindings for MyClass.

Now if you apply the fix above run the same three commands, the console output will be:

> Task :lib:compileKotlin FROM-CACHE
[...]
Task ':lib:compileKotlin' is not up-to-date because:
  Output property '$1' file [...]/lib/build/anvil/src-gen-main/anvil has been removed.
  Output property '$1' file [...]/lib/build/anvil/src-gen-main/anvil/hint has been removed.
  Output property '$1' file [...]/lib/build/anvil/src-gen-main/anvil/hint/multibinding has been removed.
Loaded cache entry for task ':lib:compileKotlin' with cache key 0fae2f2f63a340d0025ad0be3edba1fc

Notice that the Anvil-generated code is now listed as an output, and it has now been restored. That means compilation can continue as intended.

Handling Incremental Changes

Incremental builds in Gradle and Kotlin both monitor outputs.

Without the fix from above, if you comment out the annotation in MyClass.kt and run compileKotlin again, the console output will be:

> Task :lib:compileKotlin
Transforming annotations-2.4.8-1-8.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-jdk8-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-jdk7-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-common-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming annotations-13.0.jar with ClasspathEntrySnapshotTransform
Build cache key for task ':lib:compileKotlin' is d803204ae4acb428508c261acc25ed20
Task ':lib:compileKotlin' is not up-to-date because:
  Input property 'sources' file [...]/lib/src/main/kotlin/com/foo/MyClass.kt has changed.
file or directory '[...]/lib/build/kotlin/compileKotlin/local-state', not found
file or directory '[...]/lib/src/main/java', not found
file or directory '[...]/lib/src/main/java', not found
file or directory '[...]/lib/src/main/java', not found
Using Kotlin/JVM incremental compilation
Kotlin source files: [...]/lib/src/main/kotlin/com/foo/MyClass.kt
Java source files: 
Script source files: 
Script file extensions: 
[KOTLIN] Kotlin compilation 'jdkHome' argument: [...]/azul-17-ARM64/zulu-17.jdk/Contents/Home
i: found daemon on port 17040 (1855766 ms old), trying to connect
i: connected to the daemon
Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0, 3], reportSeverity=2, requestedCompilationResults=[0], kotlinScriptExtensions=[]), areFileChangesKnown=true, modifiedFiles=[[...]/lib/src/main/kotlin/com/foo/MyClass.kt], deletedFiles=[], classpathChanges=NoChanges, workingDir=[...]/lib/build/kotlin/compileKotlin/cacheable, multiModuleICSettings=MultiModuleICSettings(buildHistoryFile=[...]/lib/build/kotlin/compileKotlin/local-state/build-history.bin, useModuleDetection=false), usePreciseJavaTracking=true, outputFiles=[[...]/lib/build/classes/kotlin/main, [...]/lib/build/kotlin/compileKotlin/cacheable, [...]/lib/build/kotlin/compileKotlin/local-state])
Stored cache entry for task ':lib:compileKotlin' with cache key d803204ae4acb428508c261acc25ed20

Here’s the important part:

IncrementalCompilationOptions(
  [...] 
  outputFiles=[
    [...]/lib/build/classes/kotlin/main,
    [...]/lib/build/kotlin/compileKotlin/cacheable,
    [...]/lib/build/kotlin/compileKotlin/local-state
  ]
)

Kotlin is not watching the Anvil-generated code as part of its incremental compilation logic. That’s why we’re here.

Now if I:

1. apply the fix above
2. restore the annotation in MyClass.kt
3. re-run ./gradlew compileKotlin
4. comment out the annotation in MyClass.kt again
5. re-run ./gradlew compileKotlin -i

The output is now:

> Task :lib:compileKotlin
Transforming annotations-2.4.8-1-8.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-jdk8-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-jdk7-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming kotlin-stdlib-common-1.8.22.jar with ClasspathEntrySnapshotTransform
Transforming annotations-13.0.jar with ClasspathEntrySnapshotTransform
Build cache key for task ':lib:compileKotlin' is fe802a676b1a8030fe43e7a00c11c09c
Task ':lib:compileKotlin' is not up-to-date because:
  Input property 'sources' file [...]/lib/src/main/kotlin/com/foo/MyClass.kt has changed.
file or directory '[...]/lib/build/kotlin/compileKotlin/local-state', not found
file or directory '[...]/lib/src/main/java', not found
file or directory '[...]/lib/src/main/java', not found
file or directory '[...]/lib/src/main/java', not found
Using Kotlin/JVM incremental compilation
Kotlin source files: [...]/lib/src/main/kotlin/com/foo/MyClass.kt
Java source files: 
Script source files: 
Script file extensions: 
[KOTLIN] Kotlin compilation 'jdkHome' argument: [...]/azul-17-ARM64/zulu-17.jdk/Contents/Home
i: found daemon on port 17040 (2456952 ms old), trying to connect
i: connected to the daemon
Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0, 3], reportSeverity=2, requestedCompilationResults=[0], kotlinScriptExtensions=[]), areFileChangesKnown=true, modifiedFiles=[[...]/lib/src/main/kotlin/com/foo/MyClass.kt], deletedFiles=[], classpathChanges=NoChanges, workingDir=[...]/lib/build/kotlin/compileKotlin/cacheable, multiModuleICSettings=MultiModuleICSettings(buildHistoryFile=[...]/lib/build/kotlin/compileKotlin/local-state/build-history.bin, useModuleDetection=false), usePreciseJavaTracking=true, outputFiles=[[...]/lib/build/anvil/src-gen-main/anvil, [...]/lib/build/classes/kotlin/main, [...]/lib/build/kotlin/compileKotlin/cacheable, [...]/lib/build/kotlin/compileKotlin/local-state])
Stored cache entry for task ':lib:compileKotlin' with cache key fe802a676b1a8030fe43e7a00c11c09c

And here’s that same important part:

IncrementalCompilationOptions(
  [...] 
  outputFiles=[
    [...]/lib/build/anvil/src-gen-main/anvil,
    [...]/lib/build/classes/kotlin/main,
    [...]/lib/build/kotlin/compileKotlin/cacheable,
    [...]/lib/build/kotlin/compileKotlin/local-state
  ]
)

KGP is populating its incremental compilation outputs based upon the outputs of the KotlinCompile task. Simply adding the directory to outputs is enough to fix the behavior.

Permanent Fix?

I’ll do a quick update to AnvilPlugin and get a snapshot out tonight, and I’ll comment here when it’s available.

This issue highlights a lack of test coverage around the Gradle plugin itself. There’s currently nothing in the way of Gradle Test Kit tests. I’ll be adding the scaffolding and tests themselves within the next few days. Hopefully we can get a new release with a permanent fix out soon.

We’re currently evaluating a couple different options but the TLDR is that it’s not a simple fix and will likely require changes on both the Anvil side and Kotlin side to fully address. I’ll post an update when we have more concrete details to share.

Quick update on where we’re at with this: Currently there doesn’t appear to be a feasible way for us to completely fix this issue without a new IC API for compiler plugins. That new API is being tracked in KT-51733.

Separately, we’ve also begun working on KSP support for Anvil. KSP has the type of IC API that we need in-place already, which should help to resolve this issue. However, I can’t give an estimate on when KSP support will be ready since we are just starting and it will also depend on when Dagger finishes adding its KSP support.

In the meantime, the best known work-around is still using the old IC by setting kotlin.incremental.useClasspathSnapshot=false, as noted by @hungvietnguyen above.

You may also notice slightly fewer instances of incremental compilation issues with Kotlin 1.9.0 and Anvil 2.4.7+ thanks to KT-58289 being fixed.

Thanks for the update. Are the IC regressions in the recent versions (such as https://github.com/square/anvil/issues/710) also caused by the same issue?

Yes, as far as I can tell. I retested the use-case from #710 and observed the same behavior outlined in issue 2 around the difficulties with managing the generated output files.

That said, maybe kotlin.incremental.useClasspathSnapshot=false can be used if the performance penalty of kotlin.incremental=false is too much and you’re happy with reducing the number of incremental compilation issues rather than resolving all of them.

Thanks for expanding on my comment @hungvietnguyen, I could have been clearer about it being situationally helpful for reducing the incremental issue occurrences (as opposed to a complete work-around).

I can’t give an estimate on when KSP support will be ready since we are just starting and it will also depend on when Dagger finishes adding its KSP support.

Also adding a quick clarification on this ^: although full Anvil KSP support will depend on Dagger KSP, adding KSP support for factory generation is something we can do independently. So barring any major gotchas, that piece should some a bit sooner 🤞.

There seem to be 2 issues here:

[Fixed] Issue 1 - :app:kaptGenerateStubsDebugKotlin is UP-TO-DATE when an annotation is added/removed in :mylibrary

As explained in https://github.com/square/anvil/issues/693#issuecomment-1500455299, this is a regression in KGP 1.8.20 where the new IC (kotlin.incremental.useClasspathSnapshot) is enabled by default.

I’ve verified that the issue has been fixed in KGP 1.9.0-Beta (https://youtrack.jetbrains.com/issue/KT-58289). That is, in KGP 1.9.0-Beta, :app:kaptGenerateStubsDebugKotlin is no longer UP-TO-DATE when an annotation is added/removed in :mylibrary.

[Not Yet Fixed] Issue 2 - Anvil doesn’t generate mylibrary/build/anvil/src-gen-debug when an annotation is added in :mylibrary

Steps to reproduce:

1. Use the sample project in comment 1 (which uses KGP 1.8.0).
2. Make sure that in mylibrary/src/main/java/com/matejdro/mylibrary/LibraryMultibinds.kt, @ContributesMultibinding(AppScope::class) is removed.
3. Run ./gradlew clean :app:kaptGenerateStubsDebugKotlin (or ./gradlew clean :mylibrary:compileDebugKotlin to focus on this task).
4. In mylibrary/src/main/java/com/matejdro/mylibrary/LibraryMultibinds.kt, add @ContributesMultibinding(AppScope::class).
5. Run ./gradlew :app:kaptGenerateStubsDebugKotlin (or ./gradlew :mylibrary:compileDebugKotlin).
6. Observe that mylibrary/build/anvil/src-gen-debug is empty.
   • Side note:
     • For some reason, app/build/anvil/src-gen-debug/anvil/module/com/matejdro/multibindsanvildemo/AppComponent.kt is still correctly re-generated (it contains LibraryMultibinds), but maybe it’s just lucky, let’s focus on the fact that mylibrary/build/anvil/src-gen-debug is empty.
     • When the annotation is removed, mylibrary/build/anvil/src-gen-debug is deleted (as expected), so the bug doesn’t manifest in this scenario. (Again, this seems to be just by luck – see root cause below.)

Root cause:

• In an incremental build where the annotation is added or removed (step 5 above), com.squareup.anvil.compiler.codegen.CodeGenerationExtension#analysisCompleted is called 4 times. That’s because the Kotlin incremental compiler runs in multiple rounds (see this while loop). In this example, there are 2 rounds: Each round calls the Kotlin compiler once, and each Kotlin compiler invocation calls the com.squareup.anvil.compiler.codegen.CodeGenerationExtension#analysisCompleted method twice.
• The data sent for each round is different, and Anvil doesn’t seem to be able to handle that. In this case, mylibrary/build/anvil/src-gen-debug is correctly generated in the first round, but gets deleted in the second round, resulting in an empty directory when the task finishes.
• This happens with KGP 1.8.0, with or without the new IC (kotlin.incremental.useClasspathSnapshot), so it’s likely to be a bug in Anvil.

To sum up: It seems that Anvil currently doesn’t work well with Kotlin incremental compilation (IC). Specifically, it doesn’t handle the fact that Kotlin IC works in multiple rounds with multiple calls to the Kotlin compiler with different data each time.

Anvil currently disables Kotlin IC only for the KaptGenerateStubsTask. Maybe it should do that for the KotlinCompile task too until it can support Kotlin IC properly?

@svilen-ivanov can confirm, I received exactly the same issue when tried to launch :sample:app:assembleDebug.

Steps to reproduce:

1. Apply the changes proposed by @RBusarow both to library and app module
2. Run :sample:app:assembleDebug
3. Replace RealFatherProvider’s object provision with @Inject-constructor.
4. Run :sample:app:assembleDebug again

Error will be returned during :sample:app:kaptDebugKotlin execution:

error: [Dagger/MissingBinding] com.squareup.anvil.sample.father.FatherProvider cannot be provided without an @Provides-annotated method.

Project sample:library itself compiles without errors, but anvil/src-gen-debug directory is empty.

The sample project was very useful! I found that when @ContributesMultibinding(AppScope::class) is added to class LibraryMultibinds in :mylibrary, the new IC doesn’t detect a change because the new IC relies on the Kotlin class metadata in LibraryMultibinds.class and in this case the Kotlin class metadata doesn’t change (whether this is a bug is TBD [1]).

Because of this, the :app:kaptGenerateStubsDebugKotlin task is UP-TO-DATE. (This seems correct for this task because its output shouldn’t depend on changes to annotations of classes on the classpath.)

However, Anvil seems to rely on the :app:kaptGenerateStubsDebugKotlin task being run in order to do some work (e.g., to generate app/build/anvil/src-gen-debug/anvil/module/com/matejdro/multibindsanvildemo/AppComponent.kt which contains references to AppMultibinds and LibraryMultibinds). This can be seen at this code comment.

I guess we’ll need to figure out [1] first. From there, it will be clear whether the bug is in the new IC or in Anvil.

I can confirm the beta version works for my team after opting in to trackSourceFiles! Changes to files with a (custom) code generator applied to them no longer give any incremental build issues.

Updating my custom code generator to use GeneratedFileWithSources was easy.

Thanks to everyone who helped figure this out and resolved it. 👏

This seems fixed by v2.5.0-beta01 with trackSourceFiles on 🎉

Thanks for the workaround!

After some testing, it appears the workaround breaks UP-TO-DATE checks for ksp tasks. All ksp tasks in modules where anvil is seem to always execute, even without changes. Info from build scan points to this workaround:

It seems that ksp task extends KotlinCompile and thus causing this. I’ve managed to workaround it with yet another hack by excluding ksp task from the workaround, but I’m not sure what are the full implications of this:

tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinCompile>().configureEach {
    if (this !is KspTaskJvm && this !is KaptGenerateStubsTask) {
        val ssName = sourceSetName.get()
        val anvilSrcGenDir = layout.buildDirectory.dir("anvil/src-gen-$ssName/anvil")
        // adds the Anvil directory to the task's outputs
        outputs.dir(anvilSrcGenDir)
    }
}

In the meantime, the best known work-around is still using the old IC by setting kotlin.incremental.useClasspathSnapshot=false, as noted by @hungvietnguyen above.

I believe the workaround is to set kotlin.incremental=false in gradle.properties. This will avoid the incremental compilation issues completely because compilation will always be non-incremental. The downside is that it will slow down your build significantly.

I wouldn’t recommend setting kotlin.incremental.useClasspathSnapshot=false instead of kotlin.incremental=false. The reason is that:

• For KGP 1.9.0+, kotlin.incremental.useClasspathSnapshot=false will not help (because the issue with kotlin.incremental.useClasspathSnapshot=true – Issue 1 in this comment – has been fixed in KGP 1.9.0+).
• For KGP 1.8.22 and earlier, kotlin.incremental.useClasspathSnapshot=false only fixes Issue 1 in this comment; Issue 2 in that comment, which is more fundamental, is still not fixed.
  • That said, maybe kotlin.incremental.useClasspathSnapshot=false can be used if the performance penalty of kotlin.incremental=false is too much and you’re happy with reducing the number of incremental compilation issues rather than resolving all of them.

in this case the Kotlin class metadata doesn’t change (whether this is a bug is TBD [1]).

I’ve filed https://youtrack.jetbrains.com/issue/KT-57919/Kotlin-class-metadata-doesnt-contain-info-about-class-annotations for the Kotlin team to take a look.

@RBusarow thanks for the workaround, I see when we add the new output in the task we are producing overlapping outputs: https://ge.solutions-team.gradle.com/s/pk6mtthejsday/timeline?details=j7bq5uuamihla I tested as well the #836 branch and I’m seeing the same problem.

If you use custom Anvil code generators under your own namespace, make sure to remove the anvil namespace from the end of the build directory path: layout.buildDirectory.dir("anvil/src-gen-$ssName")

I’m using java-test-fixtures plugin. Due to unknown reason to me it has undefined sourceSetName. Here is my workaround:

   // ....
    val ssName = when {
        sourceSetName.isPresent -> sourceSetName.get()
        name == "compileTestFixturesKotlin" -> "testFixtures"
        else -> error("Cannot determine sourceSetName for task $name")
    }
   // ....

EDIT. After I brought up my project to working state, the workaround doesn’t work for me. The anvil/src-gen-test/anvil directory is still wiped out on the second build when a change (added constructor parameter) to a class that participates in multi-binding. 😦

Does anyone found a workaround that allows updating Kotlin version without disabling incremental compilation?

Hi @JoelWilcox, just wanted to follow up if multifileFacadeToParts API mentioned in KT-51733 was helpful in any way?

Also - as Dagger has just brought KSP support are there any gaps in Anvil that could be filled in by external contributors to help solving this issue?

Thanks for the update. Are the IC regressions in the recent versions (such as #710) also caused by the same issue?

Sorry folks for confusion I meant we tried this option kotlin.incremental.useClasspathSnapshot=false with false (not true), and it didn’t work for us.

This bug should only happen if you add/remove the ContributesMultibinding annotation. It should not happen if you “touch any class with ContributesMultibinding” but do not add/remove that annotation.

I bet we saw this issue even when we add a new argument (dependency) to the inject constructor, its not just adding / removing annotation but even changing the dependencies.