quarkus: JaCoCo not working with `@QuarkusIntegrationTest`s

Describe the bug

After upgrading to Quarkus 2 we saw a test coverage loss of 6% for both unit and integration tests.

I’ve tried to change how we have everything setup following https://quarkus.io/guides/tests-with-coverage (since we were still using offline instrumentation), starting with our integration tests, but I can’t get things to work right (JaCoCo complaints about classes not matching execution data). I’ve created a small reproducer (linked below) by setting up the necessary parts as in https://quarkus.io/guides/tests-with-coverage and then having the following test (since we can’t use @QuarkusIntegrationTest for a couple of reasons):

public class AppIT {

    @Test
    void myTest() throws Exception {
        new ProcessBuilder()
                .command("java", System.getProperty("quarkus.test.argLine"), "-jar", "target/quarkus-app/quarkus-run.jar")
                .inheritIO()
                .start()
                .waitFor();
    }

}

I also tried then with a REST resource and a proper @QuarkusIntegrationTest (reproducer also linked below) and got the same result: JaCoCo still complaining and showing a coverage of 0%.

Expected behavior

Can get JaCoCo working together with Quarkus, finding the right classes and reporting the right coverage.

Actual behavior

[INFO] --- jacoco-maven-plugin:0.8.7:report (report-it) @ app ---
[INFO] Loading execution data file /Users/xtaixe/dev/quarkus-reproducer/app/target/jacoco-quarkus.exec
[INFO] Analyzed bundle 'app' with 2 classes
[WARNING] Classes in bundle 'app' do not match with execution data. For report generation the same class files must be used as at runtime.

And JaCoCo reports an incorrect coverage (0% in the reproducers).

To Reproduce

quarkus-reproducer-1.zip quarkus-reproducer-2.zip

Environment (please complete the following information):

Output of `uname -a` or `ver`

Darwin xtaixe 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 x86_64

Output of `java -version`

openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment GraalVM CE 21.1.0 (build 11.0.11+8-jvmci-21.1-b05)
OpenJDK 64-Bit Server VM GraalVM CE 21.1.0 (build 11.0.11+8-jvmci-21.1-b05, mixed mode, sharing)

Quarkus version or git rev

2.0.1.Final

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: /Users/xtaixe/.mvnvm/apache-maven-3.8.1
Java version: 11.0.11, vendor: GraalVM Community, runtime: /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.1.0/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "11.4", arch: "x86_64", family: "mac"

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 25 (12 by maintainers)

Commits related to this issue

Make jacoco work with @QuarkusIntegrationTest To make this work, essentially what we do is to overwrite the build system output with the transformed bytecode See https://github.com/quarkusio/quarkus/... — committed to geoand/quarkus by geoand 3 years ago
Make jacoco work with @QuarkusIntegrationTest To make this work, essentially what we do is to overwrite the build system output with the transformed bytecode See https://github.com/quarkusio/quarkus/... — committed to geoand/quarkus by geoand 3 years ago
Mark the auto-generated RESTEasy constructor as synthetic This is mainly done in order to allow code coverage tools to skip the constructor Relates to: #18559 — committed to geoand/quarkus by geoand 3 years ago
Make jacoco work with @QuarkusIntegrationTest To make this work, essentially what we do is to overwrite the build system output with the transformed bytecode See https://github.com/quarkusio/quarkus/... — committed to geoand/quarkus by geoand 3 years ago
Make jacoco work with @QuarkusIntegrationTest To make this work, essentially what we do is to overwrite the build system output with the transformed bytecode See https://github.com/quarkusio/quarkus/... — committed to geoand/quarkus by geoand 3 years ago
Merge pull request #20319 from geoand/#18559 Make jacoco work with @QuarkusIntegrationTest — committed to quarkusio/quarkus by geoand 3 years ago
Make jacoco work with @QuarkusIntegrationTest To make this work, essentially what we do is to overwrite the build system output with the transformed bytecode See https://github.com/quarkusio/quarkus/... — committed to geoand/quarkus by geoand 3 years ago

Most upvoted comments

When I run the integration test with @QuarkusTransactionalTest, the coverage of the tested class shows as 0. Then I added the following dependencies based on the https://quarkus.io/guides/tests-with-coverage#coverage-for-integration-tests:

<dependency>
    <groupId>io.quarkus</groupId>
    <artifactId>quarkus-jacoco</artifactId>
    <version>${quarkus.version}</version>
    <scope>test</scope>
</dependency>

<quarkus.version>2.15.0.Final</quarkus.version>

The coverage report goes back to normal.

tu-yucheng on Jan 19, 2023

Hey, finally got time to properly look into this again and I got it working, but quarkus.package.write-transformed-bytecode-to-build-output seems to be incompatible with the Quarkus JaCoCo extension. When using both, JaCoCo complains again about classes not being the same and in this case it fails to report the proper coverage for classes covered by NON integration tests.

The only way I got both unit and integration tests reporting the right coverage together (on a very simple maven single module project) was by:

Using quarkus.package.write-transformed-bytecode-to-build-output.
Removing the Quarkus JaCoCo extension.
Removing <exclClassLoaders>*QuarkusClassLoader</exclClassLoaders> from the jacoco-maven-plugin configurations.

Attaching working single module project for reference: tests-with-coverage-quickstart-single-module.zip

On a multi module project with an aggregated JaCoCo report though, there’s always some incompatibilities between the classes that were unit tested and the classes that JaCoCo sees when doing the report (or the other way around if for example I tell Quarkus to not remove “unused” classes). After coming to know that the classes are copied during the package phase, I’m not sure it’s going to be possible to aggregate unit and integration test reports at all, and I have no idea why everything just works in a single module project.

cc @geoand and @stuartwdouglas in case you have some comments/suggestions or think it’s worth to create some issues.

xtaixe on Mar 17, 2022

@geoand no, sorry if that was confusing. That was the trigger to trying to migrate from offline instrumentation to use JaCoCo as explained in https://quarkus.io/guides/tests-with-coverage, but the drop was caused by something else.

The problem is that we still need to move away from offline instrumentation and I can’t get JaCoCo to work for @QuarkusIntegrationTest as explained in https://quarkus.io/guides/tests-with-coverage and there’s no sample/quickstart project showing that it works since https://github.com/quarkusio/quarkus-quickstarts/tree/main/tests-with-coverage-quickstart is outdated. If you take a look at the last reproducer which is really simple and follows https://quarkus.io/guides/tests-with-coverage, you should see that for the parts of the code that are only tested by the @QuarkusIntegrationTest JaCoCo reports a coverage of 0% (cause it says that execution and report classes don’t match).

xtaixe on Sep 9, 2021

@bvarner

Are you using @QuarkusIntegrationTests?
Are you using offline instrumentation? If not, could you explain or upload a small example (maybe based on the second reproducer attached in the description above) about how you configure the jacoco-maven-plugin to get it working and report code covered by @QuarkusIntegrationTests?

I already tried with the latest Quarkus version I haven’t been able to get JaCoco to work with @QuarkusIntegrationTests (that second reproducer still shows a coverage of 0 for the class that is only covered by a @QuarkusIntegrationTest), so I’d really appreciate some guidance on this.

xtaixe on Sep 3, 2021