extensions: firestore-bigquery-export fs-bq-import-collection doesn't allow {widecard} path

[READ] Step 1: Are you in the right place?

Issues filed here should be about bugs for a specific extension in this repository. If you have a general question, need help debugging, or fall into some other category use one of these other channels:

For general technical questions, post a question on StackOverflow with the firebase tag.
For general Firebase discussion, use the firebase-talk google group.
To file a bug against the Firebase Extensions platform, or for an issue affecting multiple extensions, please reach out to Firebase support directly.

[REQUIRED] Step 2: Describe your configuration

Extension name: firestore-bigquery-export
Configuration values (redact info where appropriate):
- fs-bq-import-collection

[REQUIRED] Step 3: Describe the problem

Steps to reproduce:

Clone this repo.
Go to extensions/firestore-bigquery-export/scripts/import and install dependencies (npm install)
Execute npm run import
Pass the ${PROJECT_ID}
Try pass a widecard path as described on docs, like chats/{chatid}/posts
Get an error: >> The collection path must only contain letters or spaces

Expected result

Allow widecard paths to import.

Actual result

Get an error: >> The collection path must only contain letters or spaces

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 23
Comments: 24 (5 by maintainers)

Most upvoted comments

@kienner-philippe and @garciapd, I was able to get around this by making my own copy of the script and modifying the query to use collectionGroups.

Changed this: https://github.com/firebase/extensions/blob/next/firestore-bigquery-export/scripts/import/src/index.ts#L188

To this:

    let query = firebase
      .firestore()
      .collectionGroup(sourceCollectionPath)
      .limit(batch);

Added these to my package.json:

"devDependencies": {
    "@firebaseextensions/firestore-bigquery-change-tracker": "^1.0.0",
    "@google-cloud/bigquery": "^4.4.0",
    "inquirer": "^7.0.0",
}

Then I just run my local version of the script using ts-node with something like this:

GOOGLE_APPLICATION_CREDENTIALS="./secrets/service-account-key.json" \
    npx ts-node ./bin/firestore-bigquery-export-backfill.ts  # my local version of the file

when the script asks:

What is the path of the the Cloud Firestore Collection you would like to import from?

Give it the sub-collection name for the collection group query. posts

+10

MattReidArnold on Dec 23, 2019

please support sub collections!!

liberathor on Sep 16, 2020

This is a great tool but definitely need a way to be able to do wildcards and collection groups would be good to. I have a small dataset so for now I’m going to cheat and just run updates on my existing records.

jttaynton on Oct 25, 2019

Hey, is this feature still not being worked on? I have a fix for it and can open a pull request if it’s not been implemented yet.

osheari1 on Mar 6, 2020

I don’t think that will work, @crablab. Wildcards are not supported in Firestore collection queries. You would have to target each collection individually.

russellwheatley on Aug 3, 2020

@russellwheatley thanks! Took care of it the way you recommended:)

Infinitism on Jun 16, 2020

wild card is one thing, unable to import sub collection is a bigger issue. I wasn’t able to implement the hacking script method, could you please support sub collections?

fasteater on May 3, 2020

@osheari1 If you have a fix for it then you’re more than welcome to open a PR. I closed the issue because wildcard paths are not yet supported as a Firestore query, and a work around solution was merged in.

russellwheatley on Mar 9, 2020

Thanks for the feedback. This is not yet supported because we do not yet have a way to translate wild card path into a Firestore query. Wildcard are supported in triggers by default.

A low-hanging fruit might be adding support for collection group in the import script. Will that solve your use case?

fredzqm on Oct 1, 2019