extensions: Firebase Biq Query extension - Exceeded rate limits: too many api requests per user per method for this user_method

Configuration firebase-admin: 9.12.0 firebase-functions: 3.24.1 firebase/firestore-bigquery-export@0.1.24

Issue I have the firebase extension for streaming data to Big Query installed https://extensions.dev/extensions/firebase/firestore-bigquery-export.

Each month I run a job to import data into my Firestore collection in batches. This month I imported 2706 rows but only 2646 made it into Big Query (60 less).

I am getting the following errors from the extension:

Error: Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas Error: Process exited with code 16 at process. I contacted Firebase support and they suggested I upgrade to the latest firebase admin and function packages but these have breaking changes. Updating the latest version of firebase-admin gave me errors. I have not got any more help from them and it is still happening for multiple collections.

The options I see are:

1. Update to the latest firebase-admin and firebase-functions packages and change my code to work with the breaking changes. I think this is unlikely to help.
2. Update the firebase extension to the latest version from 0.1.24 to 0.1.29 which now includes a flag called “Use new query syntax for snapshots” which can be turned on. I can’t find much information about this. I have now done this but will not find out until next month if it works as it is hard to replicate in dev.
3. Increase the Big Query quota somehow.
4. Slow down the data being entered into Firestore or add it daily/weekly rather than monthly.

If I look in the GCP logs using the query:

protoPayload.status.code ="7"
protoPayload.status.message: ("Quota exceeded" OR "limit")

Here is my code in Nodejs:

 const platformFeesCollectionPath = `platformFees`;
  const limit = 500;
  let batch = db.batch();
  let totalFeeCount = 0;
  let counter = 0;

  for (const af of applicationFees) {
    const docRef = db.collection(platformFeesCollectionPath).doc();
    batch.set(docRef, { ...af, dateCreated: getTimestamp(), dateModified: getTimestamp() })

    counter++;
    if (counter === limit || counter === applicationFees.length) {
      await batch.commit();
      console.log(`Platform fees batch run for ${counter} platform fees`);
      batch = db.batch();
      totalFeeCount = totalFeeCount + counter;
      counter = 0;
    }
  }

  if (applicationFees.length > limit) {
    // Need this commit if there are multiple batches as the applicationFees.length does not work
    await batch.commit();
    totalFeeCount = totalFeeCount + counter;
  }
  
  if (counter > 0) {
    console.log(`Platform fees batch run for ${totalFeeCount} platform fees`);
  }

https://stackoverflow.com/questions/75044455/firebase-biq-query-extension-exceeded-rate-limits-too-many-api-requests-per-u