extensions: Error with latest version of Big Query extension 0.1.33

[REQUIRED] Step 2: Describe your configuration

Stream Firestore to BigQuery firebase/firestore-bigquery-export@0.1.33

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Install the extension with the latest version and configure. Add data to collections to sync with BQ.

Expected result

No errors and all data comes through to BQ.

Actual result

Errors and all data comes through to BQ.

Error when mirroring data to BigQuery FirebaseFunctionsError: The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/hivepass-prod/locations/us-east1/queues/ext-firestore-bigquery-dynamic-links-syncBigQuery"


Error when mirroring data to BigQuery FirebaseFunctionsError: The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/hivepass-prod/locations/us-east1/queues/ext-firestore-bigquery-dynamic-links-syncBigQuery" (or the resource may not exist).
    at FunctionsApiClient.toFirebaseError (/workspace/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:227:16)
    at /workspace/node_modules/firebase-admin/lib/functions/functions-api-client-internal.js:88:24
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /workspace/lib/index.js:106:9 {
  errorInfo: {
    code: 'functions/permission-denied',
    message: 'The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/hivepass-prod/locations/us-east1/queues/ext-firestore-bigquery-dynamic-links-syncBigQuery" (or the resource may not exist).'
  },
  codePrefix: 'functions'
}

About this issue

Original URL
State: closed
Created a year ago
Reactions: 4
Comments: 24 (9 by maintainers)

Most upvoted comments

We faced the exact same problem, our original extension was installed to EU. @dackers86 These steps might help you to reproduce.

You need an old version of extension e.g.: 0.1.24 if I am not mistake this was our old version.
Install it to EU multiregion and function location to europe-west1
Upgrade the extension without modifying any data.
Your function should be in europe-west1 but it will be in us-central1, and the problem is that, it wants to use a task queue in europe-west1 but the queue is in us-central1 so it can’t use it.

The only solution (I didn’t try much) that worked for me is the reinstall of the extension.

Also I don’t know if this extension is using it or not, but firebase-tools has default location related fix in their changelog mentioned.

lmesz on Aug 7, 2023

I have a similar problem and after the last update my Cloud Functions location suddenly changed from Frankfurt Germany to Iowa. This is an irreversible operation and cannot be manually changed from my side any more. I guess that the right problems arise from this fact.

All my export functions are now located in us-central1

I think this is also a privacy issue and should be immediately fixed.

As I have a multitude of those export functions redeploying them will take some time.

luckyhandler on Jul 25, 2023

I just tried that. Uninstalling and installing the extension again fixes the problem. Lucky I just tested on non critical collections first. I will wait for a solution before moving my high transaction and critical collections to v0.1.33 as the down time may cause issues. Thanks for the real-time help!

madmacc on Jul 17, 2023

Ok, thanks. We can try to replicate this scenario to see if I can find a solution.

Installing a new instance, could potentially be a solution in the meantime?

dackers86 on Jul 17, 2023

Got it at: https://console.cloud.google.com/functions/details/us-central1/ext-firestore-bigquery-dynamic-links-syncBigQuery?env=gen1&project=hivepass-prod&authuser=3

madmacc on Jul 17, 2023