google-cloud-java: Dependency conflicts with Apache Beam Cloud Dataflow runner
Hi, I’m running into the following exception when using Firestore:
java.lang.NoSuchMethodError: com.google.auth.oauth2.ServiceAccountCredentials.getProjectId()Ljava/lang/String;
at com.google.firebase.FirebaseApp.getProjectId(FirebaseApp.java:307)
at com.google.firebase.ImplFirebaseTrampolines.getProjectId(ImplFirebaseTrampolines.java:43)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:32)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:26)
at com.google.firebase.cloud.FirestoreClient$FirestoreClientService.<init>(FirestoreClient.java:81)
at com.google.firebase.cloud.FirestoreClient.getInstance(FirestoreClient.java:71)
at com.google.firebase.cloud.FirestoreClient.getFirestore(FirestoreClient.java:64)
at com.google.firebase.cloud.FirestoreClient.getFirestore(FirestoreClient.java:52)
My pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
<dependencies>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>com.intellij</groupId>
<artifactId>annotations</artifactId>
<version>12.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>24.0-jre</version>
</dependency>
<dependency>
<groupId>com.google.auto.value</groupId>
<artifactId>auto-value</artifactId>
<version>1.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.9.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>5.0.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-params</artifactId>
<version>5.0.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.11.2</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-direct-java</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.2.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.google.firebase</groupId>
<artifactId>firebase-admin</artifactId>
<version>5.8.0</version>
</dependency>
</dependencies>
</project>
and output of mvn dependency:tree
com.canigraduate.uchicago:scraper:jar:1.0-SNAPSHOT
+- org.slf4j:slf4j-jdk14:jar:1.7.25:compile
| \- org.slf4j:slf4j-api:jar:1.7.25:compile
+- com.intellij:annotations:jar:12.0:compile
+- com.google.guava:guava:jar:24.0-jre:compile
| +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
| +- org.checkerframework:checker-compat-qual:jar:2.0.0:compile
| +- com.google.errorprone:error_prone_annotations:jar:2.1.3:compile
| +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
| \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.14:compile
+- com.google.auto.value:auto-value:jar:1.5:provided
+- org.assertj:assertj-core:jar:3.9.0:test
+- org.junit.jupiter:junit-jupiter-engine:jar:5.0.3:test
| +- org.apiguardian:apiguardian-api:jar:1.0.0:test
| +- org.junit.platform:junit-platform-engine:jar:1.0.3:test
| | +- org.junit.platform:junit-platform-commons:jar:1.0.3:test
| | \- org.opentest4j:opentest4j:jar:1.0.0:test
| \- org.junit.jupiter:junit-jupiter-api:jar:5.0.3:test
+- org.junit.jupiter:junit-jupiter-params:jar:5.0.3:test
+- org.hamcrest:hamcrest-all:jar:1.3:test
+- org.jsoup:jsoup:jar:1.11.2:compile
+- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
| +- com.fasterxml.jackson.core:jackson-core:jar:2.8.9:compile
| +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9:compile
| +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.9:compile
| +- org.apache.avro:avro:jar:1.8.2:compile
| | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
| | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
| | +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
| | +- org.apache.commons:commons-compress:jar:1.8.1:compile
| | \- org.tukaani:xz:jar:1.5:compile
| +- org.xerial.snappy:snappy-java:jar:1.1.4:compile
| \- joda-time:joda-time:jar:2.4:compile
+- org.apache.beam:beam-runners-direct-java:jar:2.2.0:compile
+- org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.2.0:compile
| +- org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.2.0:compile
| | +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
| | \- com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0:compile
| +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
| | +- com.google.protobuf:protobuf-java:jar:3.2.0:compile
| | +- io.grpc:grpc-core:jar:1.2.0:compile
| | | +- io.grpc:grpc-context:jar:1.2.0:compile
| | | \- com.google.instrumentation:instrumentation-api:jar:0.3.0:compile
| | +- io.grpc:grpc-protobuf:jar:1.2.0:compile
| | | +- com.google.protobuf:protobuf-java-util:jar:3.2.0:compile
| | | \- io.grpc:grpc-protobuf-lite:jar:1.2.0:compile
| | \- io.grpc:grpc-stub:jar:1.2.0:compile
| +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
| | +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.2.0:compile
| | +- com.google.apis:google-api-services-bigquery:jar:v2-rev355-1.22.0:compile
| | +- com.google.api:gax-grpc:jar:0.20.0:compile
| | | +- com.google.api:gax:jar:1.3.1:compile
| | | \- org.threeten:threetenbp:jar:1.3.3:compile
| | +- com.google.cloud:google-cloud-core-grpc:jar:1.2.0:compile
| | +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
| | +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18:compile
| | +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18:compile
| | | \- com.google.api.grpc:proto-google-iam-v1:jar:0.1.18:compile
| | +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0:compile
| | | +- com.google.http-client:google-http-client-protobuf:jar:1.20.0:compile
| | | \- com.google.http-client:google-http-client-jackson:jar:1.20.0:compile
| | +- com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0:compile
| | | \- com.google.api.grpc:grpc-google-common-protos:jar:0.1.0:compile
| | +- io.grpc:grpc-auth:jar:1.2.0:compile
| | +- io.grpc:grpc-netty:jar:1.2.0:compile
| | | +- io.netty:netty-codec-http2:jar:4.1.8.Final:compile (version selected from constraint [4.1.8.Final,4.1.8.Final])
| | | \- io.netty:netty-handler-proxy:jar:4.1.8.Final:compile
| | | \- io.netty:netty-codec-socks:jar:4.1.8.Final:compile
| | +- io.grpc:grpc-all:jar:1.2.0:compile
| | | +- io.grpc:grpc-okhttp:jar:1.2.0:compile
| | | | +- com.squareup.okhttp:okhttp:jar:2.5.0:compile
| | | | \- com.squareup.okio:okio:jar:1.6.0:compile
| | | \- io.grpc:grpc-protobuf-nano:jar:1.2.0:compile
| | | \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:compile
| | +- com.google.cloud:google-cloud-core:jar:1.0.2:compile
| | +- com.google.cloud:google-cloud-spanner:jar:0.20.0-beta:compile
| | | +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11:compile
| | | +- com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11:compile
| | | | \- com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11:compile
| | | \- junit:junit:jar:4.12:compile
| | | \- org.hamcrest:hamcrest-core:jar:1.3:compile
| | +- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile
| | +- com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0-pre3:compile
| | | +- commons-logging:commons-logging:jar:1.2:compile
| | | +- com.google.auth:google-auth-library-appengine:jar:0.7.0:compile
| | | \- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
| | +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26:compile
| | +- com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9:compile
| | \- com.google.api.grpc:proto-google-common-protos:jar:0.1.9:compile
| +- com.google.api-client:google-api-client:jar:1.22.0:compile
| | \- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
| +- com.google.http-client:google-http-client:jar:1.22.0:compile
| | \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
| | +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
| | \- commons-codec:commons-codec:jar:1.3:compile
| +- com.google.http-client:google-http-client-jackson2:jar:1.22.0:compile
| +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev213-1.22.0:compile
| +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
| +- com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
| +- com.google.auth:google-auth-library-credentials:jar:0.7.1:compile
| +- com.google.auth:google-auth-library-oauth2-http:jar:0.7.1:compile
| \- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
| +- com.google.api-client:google-api-client-java6:jar:1.20.0:compile
| +- com.google.api-client:google-api-client-jackson2:jar:1.20.0:compile
| \- com.google.oauth-client:google-oauth-client-java6:jar:1.20.0:compile
\- com.google.firebase:firebase-admin:jar:5.8.0:compile
+- com.google.api-client:google-api-client-gson:jar:1.22.0:compile
| \- com.google.http-client:google-http-client-gson:jar:1.22.0:compile
| \- com.google.code.gson:gson:jar:2.1:compile
+- com.google.api:api-common:jar:1.2.0:compile
+- com.google.cloud:google-cloud-storage:jar:1.7.0:compile
| \- com.google.cloud:google-cloud-core-http:jar:1.7.0:compile
| \- com.google.http-client:google-http-client-appengine:jar:1.22.0:compile
+- com.google.cloud:google-cloud-firestore:jar:0.25.0-beta:compile
| \- com.google.api.grpc:proto-google-cloud-firestore-v1beta1:jar:0.1.20:compile
+- org.json:json:jar:20160810:compile
+- io.netty:netty-codec-http:jar:4.1.14.Final:compile
| \- io.netty:netty-codec:jar:4.1.14.Final:compile
+- io.netty:netty-handler:jar:4.1.14.Final:compile
| \- io.netty:netty-buffer:jar:4.1.14.Final:compile
| \- io.netty:netty-common:jar:4.1.14.Final:compile
\- io.netty:netty-transport:jar:4.1.14.Final:compile
\- io.netty:netty-resolver:jar:4.1.14.Final:compile
It looks like this is due to Beam pulling in old dependencies that aren’t sufficient for firebase-admin
, so I tried pinning
<dependency>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
<version>0.9.0</version>
</dependency>
which resulted in #2496:
java.lang.NoClassDefFoundError: com/google/api/gax/rpc/ClientSettings
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
so I tried adding
<dependency>
<groupId>com.google.api</groupId>
<artifactId>gax</artifactId>
<version>1.19.0</version>
</dependency>
<dependency>
<groupId>com.google.api</groupId>
<artifactId>gax-grpc</artifactId>
<version>1.19.0</version>
</dependency>
but this results in
java.lang.NoClassDefFoundError: com/google/api/gax/rpc/TransportProvider
at com.google.cloud.firestore.FirestoreOptions.<clinit>(FirestoreOptions.java:41)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:37)
at com.google.firebase.cloud.FirestoreClient.<init>(FirestoreClient.java:26)
at com.google.firebase.cloud.FirestoreClient$FirestoreClientService.<init>(FirestoreClient.java:81)
at com.google.firebase.cloud.FirestoreClient.getInstance(FirestoreClient.java:71)
at which point I’m sort of stuck, since I thought this was provided by the gax-grpc
artifact. I’ve also tried putting firebase-admin
before all the beam dependencies, however this results in similar version conflicts…
Is there an easier way to resolve the dependency conflict between Beam and firebase-admin?
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Reactions: 2
- Comments: 21 (8 by maintainers)
Hi, @kevmo314 , I have carefully reviewed your issue and spend some energy to analyze your dependency tree. For the first issue “java.lang.NoSuchMethodError: java.lang.NoSuchMethodError: com.google.auth.oauth2.ServiceAccountCredentials.getProjectId()”, it caused by the duplicate library “com.google.auth:google-auth-library-oauth2-http”. Your project introduced three versions of this library, i.e., 0.7.1, 0.7.0, and 0.8.0. Of them, 0.7.1 is the acturally loaded one and 0.8.0 is the correct version really required by your code.
When you tried to solve the problem at first time, a new version 0.9.0 was forcedly introduced, which exacerbated the problem. So maybe the right way to work around this problem is that adding the following patch in your original pom file:
For the second problem “java.lang.NoClassDefFoundError: com/google/api/gax/rpc/ClientSettings:”, You directly introduced com.google.api:gax 1.19.0 (pretty new for your project), however, the older version exists in the dependency tree, as a result, maybe com.google.api:gax 1.9.0 is a better choice. ^_^
You can add the following patch in your pom.xml:
Then the problem may be disappeared. Hope my answer can help you.
For posterity, if I put
firebase-admin
first, then I end up with a conflict onio.grpc:grpc-core
as Beam seems to require 1.5.0. Pinning that to 1.9.0 as requested byfirebase-admin
results inhttps://github.com/apache/beam/pull/5988 will bring Beam up to date, and it has passed tests.