firebase-android-sdk: Crash: java.lang.AbstractMethodError in com.google.firebase.installations.remote.FirebaseInstallationServiceClient.readGenerateAuthTokenResponse

[REQUIRED] Step 2: Describe your environment

• Android Studio version: Bumblebee 2021.1.1 Patch 2
• Firebase Component: Cloud Messaging
• Component version: 23.0.0

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

We rebuilt our app to remove the android.permission.ACCESS_BACKGROUND_LOCATION permission.

Ever since then, we see the following crash in the Play Console:

java.lang.AbstractMethodError: 
  at com.google.firebase.installations.remote.FirebaseInstallationServiceClient.readGenerateAuthTokenResponse (FirebaseInstallationServiceClient.java:569)
  at com.google.firebase.installations.remote.FirebaseInstallationServiceClient.generateAuthToken (FirebaseInstallationServiceClient.java:421)
  at com.google.firebase.installations.FirebaseInstallations.fetchAuthTokenFromServer (FirebaseInstallations.java:566)
  at com.google.firebase.installations.FirebaseInstallations.doNetworkCallIfNecessary (FirebaseInstallations.java:390)
  at com.google.firebase.installations.FirebaseInstallations.lambda$doRegistrationOrRefresh$2 (FirebaseInstallations.java:377)
  at com.google.firebase.installations.FirebaseInstallations.$r8$lambda$VZq1h0TBcWNH8Y5yY86ujrFFyLo (FirebaseInstallations.java)
  at com.google.firebase.installations.FirebaseInstallations$$InternalSyntheticLambda$0$8f6250a76dc84afdee54bd79d6c6b27858a3db00ee2f9ff4dae9d6825fe4cbe4$0.run$bridge (FirebaseInstallations.java:18)
  at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1167)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:641)
  at java.lang.Thread.run (Thread.java:920)

Across hundreds of different devices/OS versions/etc.

It’s possible that rebuilding the app picked up some updated dependencies somewhere along the way.

No one on our team can reproduce these crashes, across multiple devices/OS versions/etc. Since we can’t get a repro, and the crash is in a Google library, we’re not sure what steps to take next. We’ve just been trying different build tool versions, compile SDKs, etc etc.

We don’t see the crashes in the Play Console pre-launch. We don’t see the crashes on Firebase Test Lab with the following configurations:

Oriole, API Level 31 Pixel 5e, API Level 30 Pixel 4, API Level 29 Pixel 3, API Level 28 Nexus 5, Virtual, API Level 19

Our configuration was (when our app worked without crashes):

Build Tools: 30.0.2 Min SDK: 16 Target SDK: 30 Compile SDK: 30

Dependencies:

com.google.android.gms:play-services-auth:17.+
com.google.firebase:firebase-messaging:20.+

We rebuilt the app with the exact same dependencies etc, but it’s possible something changed in the dependency tree.

We’ve since updated all the dependencies to the latest versions, so our build looks like:

Build Tools: 32.0.0 Min SDK: 19 Target SDK: 30 Compile SDK: 31

Dependencies:

com.google.android.gms:play-services-auth:20.+
com.google.firebase:firebase-messaging:23.+

We still see the crash in Play Console.

Please… help.

Relevant Code:

      FirebaseMessaging.getInstance().getToken().addOnCompleteListener(task -> {
        if (!task.isSuccessful()) {
          return;
        }

        String token = task.getResult();

        registerWithC2DMServer(token);
      });