grpc-java: gRPC Java is not usable from Java 9 modules

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.6.1

What JVM are you using (`java -version`)?

java version "9"
Java(TM) SE Runtime Environment (build 9+181)
Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

What did you do?

Java 9 allows users to depend on older, non-modularized versions of the libraries by “converting” them to automatic modules. For example, when Maven dependencies on grpc are configured correctly, Java 9 allows me to do the following:

module myapp.foo {
    exports com.myapp.foo;
    requires grpc.core;
}

This allowed me to use classes from the grpc-core within my Java 9 module, but unfortunately it wouldn’t compile:

ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.2:compile (default-compile) on project myapp.foo: Compilation failure: Compilation failure: 
[ERROR] the unnamed module reads package io.grpc from both grpc.core and grpc.context
[ERROR] module grpc.context reads package io.grpc from both grpc.core and grpc.context
[ERROR] module guava reads package io.grpc from both grpc.core and grpc.context
[ERROR] module error.prone.annotations reads package io.grpc from both grpc.core and grpc.context
[ERROR] module jsr305 reads package io.grpc from both grpc.core and grpc.context
[ERROR] module instrumentation.api reads package io.grpc from both grpc.core and grpc.context
[ERROR] module opencensus.api reads package io.grpc from both grpc.core and grpc.context
[ERROR] module grpc.core reads package io.grpc from both grpc.core and grpc.context

The issue is that Java 9 does not support split packages across modules and this is exactly what’s happening here, as io.grpc package exists in both grpc-core and grpc-context, and to make things worse both grpc-core and grpc-stub have transitive dependency on grpc-context.

I’ve tried excluding grpc-context from both modules using Maven exclusions, which allowed me to compile successfully, as I don’t have any direct dependencies on grpc-context. However, I was not able to run the test server, because of the missing Context class:

java.lang.NoClassDefFoundError: io/grpc/Context
        at io.grpc.internal.AbstractServerImplBuilder.build(AbstractServerImplBuilder.java:188)
        at io.grpc.testing.GrpcServerRule.before(GrpcServerRule.java:133)

There are several possible solutions, some better than the others:

Merge classes from grpc-context into grpc-core and leave empty/dummy grpc-context module around for backwards compatibility (although most people probably do not depend on it directly).
Do the same as above, but get rid of unnecessary grpc-context module.
Rename the io.grpc package in grpc-context to io.grpc.context, which would eliminate split package issue, but would break existing code that uses classes from the current location.

In any case, I’m happy to help do the work, but someone will need to decide which approach to take.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 48
Comments: 44 (16 by maintainers)

Commits related to this issue

PoC at Dead End for now Dead End because of grpc/grpc-java#3522 — committed to mmichaelis/poc-grpc by mmichaelis 5 years ago
Workaround for grpc/grpc-java#3522 In order to work around grpc/grpc-java#3522 JPMS has been disabled within this project. — committed to mmichaelis/poc-grpc by mmichaelis 5 years ago
gRPC classes (#33) * gRPC classes * PB converters WIP * PB converters WIP * PB converters WIP * PB converters WIP * PB converters WIP * Hash fixes * Fixes * Fixes * Fixes ... — committed to wavesplatform/WavesJ by Karasiq 5 years ago

Most upvoted comments

Users have requested support for Java 11 in Apache Beam and since it uses grpc it would need for grpc libraries to be compatible with Java 11. Hopefully this request helps increase the priority for fixing this.

+14

lukecwik on Aug 8, 2019

Following up on this a year later, is there any plans to actually make any required breaking changes to gRPC to fix this issue? I work for a large enterprise that has artifact on boarding requirements and gRPC sits at the core of our services. For many of these it is the lone dependencies that are not modularity compliant and we do not have the luxury of leveraging personal projects that hack around the issue.

It sounds like we have run out of ideas and have spent multiple years trying to figure a way out of this problem that doesn’t involve breaking changes somewhere. It ultimately sounds like a path just needs to be picked and go with it, understanding no matter what is done its going to break someone.

IMO worrying about breaking people classpath’s/dependency trees on upgrades should be the most minor of concerns. They are an absolute reality of Java and Maven dependency management. This already happens today across several grpc versions when guava is upgraded. The amount of times I’ve seen guava force multiple artifacts into a diamond dependency problem well surpasses the amount of gRPC projects that would be impacted by any modularity upgrades.

I’m willing to do the work, just need an SME and reviewer of gRPC to pick a path.

+10

cjohnstoniv on Sep 1, 2021

In the meantime, we have repackaged grpc-java (version 1.22.1) as a Java 9+ module as part of Helidon, in order to support our gRPC framework implementation, and have made it available on Maven Central.

This module basically merges grpc-core, grpc-context and grpc-stub into a single JAR, and adds necessary module_info.java to it.

@ejona86 This is not to say that we are not looking forward to a proper Java 9+ module support in grpc-java, which will allow us to get rid of this “necessary evil”. But we needed something now…

+10

aseovic on Oct 17, 2019

I’d be willing to help out as well, but only if the maintainers bless the effort and can ensure that there’s at least a chance of it being merged. I’d really like to solve this problem as well, as it is for myself (and I suspect a gigantic number of others) the final bit preventing us from moving to use Java modules.

dpratt on Sep 1, 2021

its now 2020/06/25, is there any progress here? Or everyone here still using ancient java 8 and planing to use until 2030+?

baha2046 on Jun 25, 2020

So now the problem still exists in Java 11. Anyone can provide a solution(even a temporary one)? Module 'xxx' reads package 'io.grpc' from both 'grpc.core' and 'grpc.context'

UkonnRa on Oct 11, 2018

This is being addressed in https://github.com/grpc/grpc-java/pull/10313 . It merges grpc-context into grpc-api. grpc-context will then be empty and will depend on grpc-api, but will exclude the unnecessary dependencies from grpc-api if you are just using io.grpc.Context. So the main difference for grpc-context-only users is the jar file increases in size, but at least no extra dependencies.

ejona86 on Jun 26, 2023

@cjohnstoniv, it is nowhere near as simple as you think. It is a land of subtle issues and hard trade-offs.

Let’s not forget that this is a self-inflicted Java module issue. Java modules enforced new restrictions on existing code. It’s also made more difficult by Maven’s weak package management. We are trying to shoe-horn an existing system into new constraints without any tools to support such a transition, while not taking anything away in the process. If Java supported type aliases or if Maven supported marking “conflicting” packages this would be a lot easier.

The simplest change is to move grpc-context into grpc-api such that grpc-context is empty and depends on grpc-api. This clearly throws dependency-sensitive grpc-context users “under the bus” as the jar size would change from ~30K to ~280K and add dependencies. It would also make grpc-api harder to maintain because we would start avoiding Guava (probably shading parts of Guava, but then we have to be sensitive to how much of Guava will be copied with each class we use). So while being simplest, this approach actually has the longest-term ramifications.

I don’t think grpc-context re-appearing on the classpath through some other dependency transitively is the end of the world.

It is bad. Really bad. Users aren’t able to debug that. The results are effectively non-deterministic. You get really weird errors that appear to have nothing to do with the actual problem. We see something similar already when Maven downgrades packages; which users can’t deal with it. But in that case there is requireUpperBoundDeps which helps a lot. With this there is no such checker available and it would require a package format change to add a checker. We have to avoid this.

But maybe with a “trick” we could use the general approach but avoid the duplication on the classpath. ~~Hacks~~ Tricks have been considered; a large variety of ~~hacks~~ tricks. But it’s really easy to miss issues that will crop up. And for the issues you find you have to consider cost/benefit while being mostly blind (that is to say, we need to weigh options on cost/benefit, but we don’t actually know the cost/benefit; we have to guess).

We did work on this a year ago, but it sort of blew up in our face as we discovered issues with the approaches that made the choice more complex and caused us to consider new options to workaround those issues. We have actually come back to it again this quarter and have made good progress. But it is clear that every solution hurts someone.

ejona86 on Aug 20, 2020

@ejona86 Although a breaking change is diametrically against keeping a major version number, if we can move from this impass with a package change in 1.x I think that will unblock a lot of developers. Any idea when this may happen?

dansiviter on Nov 12, 2020

Does recognizing it as a work around also mean that the gRPC maintainers don’t intend to solve this problem soon? It was mentioned it would be looked into the next quarter almost a year ago but doesn’t seem to have any traction since. Is there any hope for a gRPC solution soon that isn’t relying on a non-gRPC project deployed version that requires its own custom artifact management?

cjohnstoniv on Aug 19, 2020

We’ll be looking into options this quarter. Main choices are to try to reduce dependencies of grpc-api and then combine grpc-context with grpc-api or to do a trick like com.google.guava:listenablefuture’s empty version 9999.0.

ejona86 on Oct 16, 2019

@aperrot42 you can take a look at https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md specifically https://github.com/grpc/proposal/blob/master/P5-jdk-version-support.md#rationale . As part of dropping support for Java8 may be Java9 modules will be supported?

sanjaypujare on Feb 9, 2022

Did not find a simple a suitable way to import gRPC to our current java projects (we have decided to use modern java and modular builds). Is there any chance that official gRPC client side libraries & reps get updated to work with java modules ? This is an important decision architecture-wise for us as it directs our adoption of gRPC.

aperrot42 on Feb 9, 2022

@adriancole https://github.com/adriancole the issue isn’t that it’s in the io.grpc realm, its that the same package is split over multiple jars. Making grpc-context contain io.grpc.context.Context would solve this.

right, but what I meant (and didn’t say) is that if we are breaking the java package anyway, if there were a time to top-level the type, it would be now

codefromthecrypt on Oct 2, 2017