google-cloud-node: Cloud tasks stopped working as of yesterday: "failed to decode protobuf and create a before snapshot"

I’m using google tasks with firebase.

Emulators affected
Production affected

Firebase CLI version 13.1.0 and lower (tried 12.x)
NodeJS version: 18 or 20 both fail
@google-cloud/tasks version: 5.0.0 or 4.0.1 both fail

As soon as I include those lines of code in my nodejs project i get fatal errors:

code to include: import { CloudTasksClient } from '@google-cloud/tasks'; const tasksClient = new CloudTasksClient();

that’s enough for everything to stop working and fail with errors:

> {"severity":"ERROR","message":"Error: Failed to decode protobuf and create a before snapshot.\n at entryFromArgs (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/logger/index.js:130:19)\n at Object.error (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/logger/index.js:116:11)\n at createBeforeSnapshotFromProtobuf (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/common/providers/firestore.js:77:16)\n at createBeforeSnapshot (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:134:65)\n at makeChangedFirestoreEvent (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:172:37)\n at func (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:238:32)\n at /Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:524:16\n at runFunction (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:506:15)\n at runCloudEvent (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:523:11)\n at processBackground (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:490:16)"} ⚠ functions: TypeError: Cannot read properties of undefined (reading 'cloud') at Function.decode (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/protos/compiledFirestore.js:1529:130) at createBeforeSnapshotFromProtobuf (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/common/providers/firestore.js:73:52) at createBeforeSnapshot (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:134:65) at makeChangedFirestoreEvent (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:172:37) at func (/Users/olegdater/Projects/project/firebase-functions/node_modules/firebase-functions/lib/v2/providers/firestore.js:238:32) at /Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:524:16 at runFunction (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:506:15) at runCloudEvent (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:523:11) at processBackground (/Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:490:16) at /Users/olegdater/.nvm/versions/node/v20.11.0/lib/node_modules/firebase-tools/lib/emulator/functionsEmulatorRuntime.js:690:27 ⚠ Your function was killed because it raised an unhandled error.

About this issue

Original URL
State: closed
Created 5 months ago
Reactions: 1
Comments: 35 (15 by maintainers)

Commits related to this issue

https://github.com/googleapis/google-cloud-node/issues/5000 — committed to motorro/firebase-openai-chat-project by deleted user 4 months ago
Digest for bug 5000 https://github.com/googleapis/google-cloud-node/issues/5000 — committed to motorro/BeforeSnapshot by deleted user 3 months ago
How to reproduce https://github.com/googleapis/google-cloud-node/issues/5000 — committed to motorro/BeforeSnapshot by deleted user 3 months ago
Cloud tasks import errors fixed Refs: https://github.com/googleapis/google-cloud-node/issues/5000 — committed to motorro/firebase-ai-chat by motorro 3 months ago

Most upvoted comments

Great, I’ve been able to reproduce the error! @motorro, thank you so much for your patience!

I don’t have a fix yet, but I have noticed the error only occurs when triggering it from Firestore as opposed to calling it from the CLI:

The last two calls were done from the CLI.

I’ll update once I have a fix!

sofisl on Mar 20, 2024

@sofisl, thank you very much! Works perfect in my test project:

motorro on Mar 27, 2024

Thank you all for your help and patience! Very tricky bug to untangle, but here is the explanation:

When we released tasks in dual-format (ESM and CJS), we introduced a new directory structure, and we had some proto compilation depending on the old directory structure. This inadvertently caused the Cloud Tasks protos to be generated with a default root name rather than a unique name, which then interfered with the firestore library and caused a collision. This note describes it well, but unfortunately still a very difficult bug to diagnose.

@orgads had actually introduced a fix a month ago, but I didn’t have time to review it and didn’t realize that the two were related. I’ll merge this fix, and release a new version of the library next week. Hopefully this does the trick!

Big thanks to @olegdater @motorro and @orgads for the collaboration!

sofisl on Mar 23, 2024

will do, give me a couple of days to get to it 😃

olegdater on Feb 8, 2024

yea, works for me too now! thanks everyone and @sofisl especially !

olegdater on Mar 28, 2024

I just released 5.2.1 last night! Would anyone mind giving it a shot?

sofisl on Mar 26, 2024

Hi @sofisl @danielbankhead

Thank you for your efforts! I think there is no need to build a client. I’ve prepared a “digested” project that could help to reproduce the error. Cloud tasks client is used in FirebaseQueueTaskScheduler I’ve written a reproduction steps in a README there. Basically, you need to do the following:

Create Firebase project
Add Auth
Add Firestore
Add Functions
Clone the repository
Deploy to firebase
Go to Firebase console
Go to Firestore
Create a document in calculations collection
Check logs for onCalcCreated function

Please take a look. If you need an access to the project itself - please tell me

PS: Firestore is created at eur3 region if it helps

motorro on Mar 19, 2024

@sofisl, here’s a project you may try. It uses firebase-openai-chat dependency with @google-cloud/tasks peer dependency. The client is imported here These functions fail:

I’ve sent you a collaborator invite for your convenience. If you need an access to firebase project, please tell me.

Works with "@google-cloud/tasks": "4.0.1"

motorro on Mar 9, 2024

i will get to it sometime, sorry been busy, i remember 😃

olegdater on Mar 6, 2024

@olegdater It worked, thanks!!! I had ^4.0.0 instead of 4.0.0. Now it works.

LeandroTorresSicilia on Feb 1, 2024

now all the other functions stopped working because of this issue.

yep, all functions will stop working, simple importing import { CloudTasksClient } from '@google-cloud/tasks'; will break everything

I tried versions 5.0.0, 4.0.1, and 4.0.0 and get the same error

this is very strange, try to remove node_modules, remove functions directory etc, 4.0.0 works for me right now,

here’s my package.json maybe it will help

    @google-cloud/tasks": "4.0.0",
    "firebase-admin": "^11.11.1",
    "firebase-functions": "^4.6.0",
    
  "engines": {
    "node": "20"
  },

olegdater on Feb 1, 2024