extensions: firestore-bigquery-export failing

Extension name: firestore-bigquery-export

I have installed the Export Collections to BigQuery extension and it works fine however now I am attempting to import my existing data using firestore-bigquery-export by following the instructions here: https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md

However I am getting the error detailed below. Note: I get exactly the same error even if I enter a firebase projectid or collection that does not exist.

This may be an local environment issue, if so and this is not an appropriate place for this issue please let me know and close it.

$ node -v v12.13.0

$ npm -v 6.12.0

Steps to reproduce:

Run: npx @firebaseextensions/fs-bq-import-collection Enter the ${PROJECT_ID}, ${COLLECTION_PATH} and ${DATASET_ID} parameters as follows: my-fb-project-id, members, qmtbc_dev_export

Expected result

Collection imports successfully into BigQuery

Actual result

Error returned:

Importing data from Cloud Firestore Collection: members, to BigQuery Dataset: qmtbc_dev_export, Table: members_raw_changelog
Error unlinking journal file C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\lib/from-members-to-qmtbc-dev:qmtbc_dev_export:members_raw_changelog
after successful import: Error: ENOENT: no such file or directory, unlink 'C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\lib\from-members-to-qmtbc-dev:qmtbc_dev_export:members_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------
Error importing Collection to BigQuery: Error: EPERM: operation not permitted, unlink 'C:\Users\fullo\AppData\Roaming\npm-cache\_npx\10884\node_modules\@firebaseextensions\fs-bq-import-collection\node_modules\grpc\src\node\extension_binary\node-v64-win32-x64-unknown\grpc_node.node'

Also I looked at the script here: https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/scripts/import/src/index.ts

And the error is being generated at line 187. It looks like it may throw this error if no data is found or if can’t access the project/collection.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 5
Comments: 22 (10 by maintainers)

Most upvoted comments

Hey @cabljac!

I am currently having a issue to import export the data! I tried the techniques above, but I could not solve my problem.

I have all the permissions required - I am the owner of the project. But, I have not found any solution that works to me

My OS is an Ubuntu 22.10 and usually I do not have this kind of issue

matheusleo on Oct 16, 2023

Ok, in that case I’ll re-open for investigation.

dackers86 on Mar 15, 2021

I am still having this issue with Mac. No problem uploading schema files and having data populate against those, but I can’t get any other result for import than Error unlinking journal file '/Users/… … Error: ENOENT: no such file or directory, unlink '/Users/…

`? What is your Firebase project ID? ideliver-cloud
? What is the path of the the Cloud Firestore Collection you would like to import from? (This may, or may not, be the same Collection for which you p
lan to mirror changes.) idriver-app\events\incoming
? What is the ID of the BigQuery dataset that you would like to use? (A dataset will be created if it doesn't already exist) ideliver_app
? What is the identifying prefix of the BigQuery table that you would like to import to? (A table will be created if one doesn't already exist) incom
ing
? How many documents should the import stream into BigQuery at once? 300
Importing data from Cloud Firestore Collection: idriver-app\events\incoming, to BigQuery Dataset: ideliver_app, Table: incoming_raw_changelog
Error unlinking journal file /Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog after successful import: Error: ENOENT: no such file or directory, unlink '/Users/tim/.npm/_npx/4672/lib/node_modules/@firebaseextensions/fs-bq-import-collection/lib/from-idriver-app\events\incoming-to-ideliver-cloud_ideliver_app_incoming_raw_changelog'
---------------------------------------------------------
Finished importing 0 Firestore rows to BigQuery
---------------------------------------------------------`

Any help would be greatly appreciated

iDeliverApps on Jun 30, 2020

@Ehesp Much appreciated. It would be good to know if it actually works so I can continue to try and resolve. I have 2 other computers including a Mac that I can try on but will take a bit of time setting up.

Among other things I tried: node v8 and v10 (currently on v10). I tried using task manager to make sure no other versions on node were running and locking it. I deleted and reinstalled all my npm packages several times.

madmacc on Nov 5, 2019

Looks like a Windows permission issue. I’ll get a local Window environment running and give it a go.

Ehesp on Nov 5, 2019