google-cloud-java: PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first

It’s solved. As you suggested, I tried to confirm the content of the files (io.grpc.LoadBalancerProvider, io.grpc.NameResolverProvider) e.g.:

$ unzip -q -c target/myJar.jar META-INF/services/io.grpc.LoadBalancerProvider
io.grpc.grpclb.GrpclbLoadBalancerProvider

I didn’t place the files correctly before, and therefore they were not being included correctly… making the fix useless. To be clear, I have moved the files to the next location:

<project-root>/src/main/resources/META-INF/services/io.grpc.LoadBalancerProvider
<project-root>/src/main/resources/META-INF/services/io.grpc.NameResolverProvider

The content of the io.grpc.LoadBalancerProvider file is

io.grpc.internal.PickFirstLoadBalancerProvider

And the content of the io.grpc.NameResolverProvider file is

io.grpc.internal.DnsNameResolverProvider

@olavloite

When using kotlin gradle build, I’ve found that a little more tuning is desirable. The normal workaround of turning on the com.github.johnrengelman.shadow plugin, which has similar function to the maven shade plugin has an issue.

If you do the default config, below:

tasks.withType<ShadowJar> {
    mergeServiceFiles ()
}

it will work, but can result in a lot of undesirable files also ending up in the output jar, I’m unsure why. I ended up with 70mb of gradle internals.

However if you do this

tasks.withType<ShadowJar> {
    mergeServiceFiles {
        setPath("META-INF/services")
        include("io.grpc.*")
    }
}

You get the solution without extra pollution.

Would it be possible to get docs updated to make it easier for people to know this? Its an awkward problem to track down the first time it happens.

I finally resolved the error by modifying the files as below and enabling attemptDirectPathXds during Bigtable client creation.

io.grpc.LoadBalancerProvider

io.grpc.internal.PickFirstLoadBalancerProvider
io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider
io.grpc.util.OutlierDetectionLoadBalancerProvider
io.grpc.grpclb.GrpclbLoadBalancerProvider
io.grpc.xds.ClusterManagerLoadBalancerProvider
io.grpc.xds.ClusterResolverLoadBalancerProvider
io.grpc.xds.CdsLoadBalancerProvider
io.grpc.xds.PriorityLoadBalancerProvider
io.grpc.xds.ClusterImplLoadBalancerProvider
io.grpc.xds.WrrLocalityLoadBalancerProvider
io.grpc.xds.WeightedTargetLoadBalancerProvider
io.grpc.xds.WeightedRoundRobinLoadBalancerProvider
io.grpc.xds.RingHashLoadBalancerProvider
io.grpc.xds.LeastRequestLoadBalancerProvider
io.grpc.rls.RlsLoadBalancerProvider
io.grpc.protobuf.services.internal.HealthCheckingRoundRobinLoadBalancerProvider

io.grpc.NameResolverProvider

io.grpc.googleapis.GoogleCloudToProdExperimentalNameResolverProvider
io.grpc.googleapis.GoogleCloudToProdNameResolverProvider
io.grpc.xds.XdsNameResolverProvider
io.grpc.internal.DnsNameResolverProvider
io.grpc.grpclb.SecretGrpclbNameResolverProvider$Provider
io.grpc.netty.shaded.io.grpc.netty.UdsNameResolverProvider

This problem is caused if you build a fat/über-jar that contains all your dependencies. In those cases, you also need to include all service definitions of the jars that you are packaging. The way to achieve this depends on your choice of build system.

For Maven you should use the ServicesResourceTransformer. For Gradle you can use the ServiceFileTransformer.

These plugins will collect all the service definitions of the included jars and bundle these into the META-INF/services directory of your fat/über-jar.

I’m getting similar error while connecting to Bigtable from GKE. Tried adding the files into META-INF/servcies of my project & modifying transformer in maven-shade-plugin and issue still persists.

I logged into the k8s pod and extracted my application jar and found that below files are present.

io.grpc.LoadBalancerProvider

io.grpc.internal.PickFirstLoadBalancerProvider
io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider
io.grpc.util.OutlierDetectionLoadBalancerProvider

io.grpc.NameResolverProvider

io.grpc.internal.DnsNameResolverProvider

I’m using Kotlin with kotlin-maven-plugin & maven-shade-plugin Internally BigTable is using Maven: io.grpc:grpc-core:1.55.1.

Tried a lot of workaround with no luck. Any help here please !

Caused by: java.lang.IllegalStateException: Default config is invalid: Status{code=UNKNOWN, description=None of [grpclb] specified by Service Config are available., cause=null} at com.google.common.base.Preconditions.checkState(Preconditions.java:590) at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:642) at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:631) at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297) at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:391) at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:107) at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:85) at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:237) at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:231) at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:201) at com.google.cloud.bigtable.data.v2.stub.EnhancedBigtableStub.create(EnhancedBigtableStub.java:175) at com.google.cloud.bigtable.data.v2.BigtableDataClient.create(BigtableDataClient.java:165) at external_service.db.BigTableClientService$Companion.getBigTableClient(BigTableClientService.kt:18)

Thank you for your response guys. @olavloite indeed I am shading using gradle with com.github.jengelman.gradle.plugins:shadow:5.0.0

Is this an issue that will be fixed in the future in grpc or the google-java library or do I permanently need this workaround from now on?

And with the workaround, do you mean including these files in my own project and including in them in the final jar?

• https://github.com/grpc/grpc-java/blob/master/core/src/bazel-internal/resources/META-INF/services/io.grpc.LoadBalancerProvider
• https://github.com/grpc/grpc-java/blob/master/core/src/bazel-internal/resources/META-INF/services/io.grpc.NameResolverProvider