firebase-admin-java: unexpected/unparsable response

When calling FirebaseMessaging.getInstance().send(message), sometimes it failes with strange exception. Sometimes no errors for a day, sometime hundreds or thousands errors. Average error rate was about 3% mid december but is much higher today.

i checked Firebase Cloud Messaging Api in Google Cloud Platform console and see increasing 404 answer for Api Method google.firebase.fcm.v1.FcmService.SendMessage

I tried to understand the code and i think the http answer is not as expected. Probably this is related to the firebase server infrastructure, but sdk should not fail on parsing the error and answer some server Exception instead?.

java.lang.IllegalArgumentException: key details at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:360) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:318) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseAndClose(JsonParser.java:162) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseAndClose(JsonParser.java:144) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.firebase.messaging.FirebaseMessagingClientImpl$MessagingErrorHandler.safeParse(FirebaseMessagingClientImpl.java:308) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl$MessagingErrorHandler.createException(FirebaseMessagingClientImpl.java:292) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl$MessagingErrorHandler.createException(FirebaseMessagingClientImpl.java:282) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.AbstractHttpErrorHandler.handleHttpResponseException(AbstractHttpErrorHandler.java:57) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.ErrorHandlingHttpClient.send(ErrorHandlingHttpClient.java:108) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.ErrorHandlingHttpClient.sendAndParse(ErrorHandlingHttpClient.java:72) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl.sendSingleRequest(FirebaseMessagingClientImpl.java:127) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessagingClientImpl.send(FirebaseMessagingClientImpl.java:113) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging$1.execute(FirebaseMessaging.java:135) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging$1.execute(FirebaseMessaging.java:132) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.internal.CallableOperation.call(CallableOperation.java:36) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging.send(FirebaseMessaging.java:102) ~[firebase-admin-7.1.0.jar:?] at com.google.firebase.messaging.FirebaseMessaging.send(FirebaseMessaging.java:86) ~[firebase-admin-7.1.0.jar:?] at de.k2interactive.qeep.api.firebase.FirebaseAdminApi.sendMessage(FirebaseAdminApi.java:70) ~[classes/:?] at de.k2interactive.qeep.api.firebase.FirebaseSinkV1Impl.sendSingleMessageInt(FirebaseSinkV1Impl.java:130) ~[classes/:?] at de.k2interactive.qeep.api.notification.NotificationSink.pushFcmReactivation(NotificationSink.java:356) ~[classes/:?] at de.k2interactive.qeep.jobs.FcmReactivationJob.lambda$sendAll$0(FcmReactivationJob.java:89) ~[qeep-jobs-55.jar:?] at java.lang.Thread.run(Unknown Source) [?:?] Caused by: java.lang.IllegalArgumentException: key details, field private java.util.Map com.google.firebase.messaging.internal.MessagingServiceErrorResponse.error at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:451) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787) ~[google-http-client-1.38.0.jar:1.38.0] ... 22 more Caused by: java.lang.IllegalArgumentException: key details, field private java.util.Map com.google.firebase.messaging.internal.MessagingServiceErrorResponse.error at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:900) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseMap(JsonParser.java:679) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:783) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:451) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787) ~[google-http-client-1.38.0.jar:1.38.0] ... 22 more Caused by: java.lang.IllegalArgumentException: expected collection or array type but got class java.lang.Object at com.google.common.base.Preconditions.checkArgument(Preconditions.java:164) ~[guava-29.0-android.jar:?] at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:67) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:724) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseMap(JsonParser.java:679) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:783) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parse(JsonParser.java:451) ~[google-http-client-1.38.0.jar:1.38.0] at com.google.api.client.json.JsonParser.parseValue(JsonParser.java:787) ~[google-http-client-1.38.0.jar:1.38.0] ... 22 more