firebase-tools: Firestore Emulator exports cannot be imported on other platforms

I follow these steps to achieve import Firestore data from the windows machine on mac.

1. run firebase emulators:start --import=./mydir --export-on-exit
2. on firebastore emulator ui create a doc
3. stop emulator
4. copy all_namespaces_all_kinds.export_metadata and output-0 from your previous dir to mydir
5. run firebase emulators:start --import=./mydir --export-on-exit and now you have imported the data 😃

Hey! I just wanted to say that this issue is still happening between Ubuntu (importing) and Windows (exporting). I’m not opening another issue as it was said in this one 2433 to consolidate here, but I’ll leave info to help with the reproduction of it:

[REQUIRED] Environment info

firebase-tools: 8.16.2

Platform: Ubuntu 20.04.1 LTS (importing) and Windows 10 (exporting)

[REQUIRED] Steps to reproduce

Create an export on windows and then import it in Ubuntu.

[REQUIRED] Expected behaviour

Should be able to run with all the imports.

[REQUIRED] Actual behaviour

I get an IO error:

Exception in thread "main" com.google.cloud.datastore.core.exception.DatastoreException: /path/to/project/e2e-testing/emulator_data/minimal_rtdb_populate/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:316)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:62)
	at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)
Caused by: java.io.FileNotFoundException: /home/acro-comp/Dev/acro-companion/e2e-testing/emulator_data/minimal_rtdb_populate/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:211)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:153)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:312)
	... 2 more

Fixed in Firebase CLI v10.5.0.

This problem still happens, I started with windows and ran the repository on my mac, and got this error message. Is there any news on this?

@markgoho Thanks for bring this up. This is still on our roadmap and I’ve changed the type to bug for clarity. This is indeed a bit involved since it requires work in the Firestore Emulator (non-opensource) and also the CLI, which makes it a bit hard to prioritize. But we’re in no way pushing this “to the bottom of the pile”, nor do we have policies like fixing bugs strictly before feature requests.

It looks like the path in firestore_export/firestore_export.overall_export_metadata could simply always use forward slashes. Windows has no issues using an export from *nix. In our case we simply copied only that file from an export done on Linux to fix our export from Windows to work cross-platform.

@dex3r agreed it is disappointing, but consider the number of people using Firestore, then those using emulators, then those with exports, then those who also switch between different operating systems, and then compare that final number of affected people to the other issues that this team needs to fix that (presumably) affect a much larger group of people

and now consider there is a workaround mentioned here: #2421 (comment)

If this were your team, how would you prioritize this?

Agreed. This is killing me and my team. Basically Mac developers cannot write to the emulator DB for windows developers to read. Not to mention Mac developers have to manually copy and paste the latest data.

@yuchenshi can you and your team help prioritize this bug fix?

I just bumped into this issue (windows-macos) and I’m disappointed it got reported over a year ago an still is not fixed 😦

@michaelyfan the workaround we are still using is to export once on Mac/Linux and always keep the all_namespaces_all_kinds.export_metadata file from that original export despite any changes written to this file when exported on Windows.

So we always revert the all_namespaces_all_kinds.export_metadata after an emulators:export on Windows and just commit the remaining changed files.

This is uncomfortable and feels hacky but still works.

Exception in thread "main" com.google.cloud.datastore.core.exception.DatastoreException: /Users/markgoho/Projects/active-branch-tracker/db/exports/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:316)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.fetchEntities(ExportImportUtil.java:62)
	at com.google.cloud.datastore.emulator.firestore.CloudFirestore.main(CloudFirestore.java:89)
Caused by: java.io.FileNotFoundException: /Users/markgoho/Projects/active-branch-tracker/db/exports/firestore_export/all_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at com.google.cloud.datastore.emulator.impl.ExportImportUtil.parseBackupFile(ExportImportUtil.java:312)
	... 2 more

Still happening. @yuchenshi I’m curious why this was tagged a feature request and if that’s what has pushed this bug to the bottom of the pile.

@floogulinc That’s a great observation. I’ll be mostly likely going down that path when I get more time to work on it.

I’ve opened b/160784464 internally to track the work needed in the Firestore Emulator.

@yuchenshi so I did find one place we have a likely bug in the code: https://github.com/firebase/firebase-tools/blob/e10851b3b56a0473f4e3f337babcd4616e93f484/src/emulator/hubExport.ts#L57

That should use path.join(). But that’s actually not the source of the issue. The problem is within the Firestore export we have files like this (excuse the binary):

$ cat data/firestore_export/firestore_export.overall_export_metadata 
�mDN3�"O
M
Aall_namespaces\all_kinds\all_namespaces_all_kinds.export_metadata osamstern@samstern-macbookpro3 tmp.3ZGMX0TU 

The overall export metadata points to a path with \ because the export was made on Windows. When the Firestore Emulator tries to read that next file (all_namespaces\all_kinds\...) it gets the FileNotFoundException.

I’m actually not sure what the best path forward is here, do you know the best practice for handling this kind of thing? Also I have attached a .zip of an export with a single Firestore document made on my Windows PC that you can use for testing.

data.zip

@charlesfries Would you mind providing a bit more details like the logs with --debug and what emulators you’re using?

Also, it will be great if you can upload a sample export zipball that we can look into. Please make sure it does not contain sensitive information you don’t want to share with the public though, or you can reach out privately to Firebase Support.

Can gladly confirm that taking an export from macos and importing it on windows now work!  Thank you @yuchenshi et al. !! ❤️ 👍 🍾

I follow these steps to achieve import Firestore data from the windows machine on mac.

1. run `firebase emulators:start --import=./mydir --export-on-exit `

2. on firebastore emulator ui create a doc

3. stop emulator

4. copy  `all_namespaces_all_kinds.export_metadata` and  `output-0` from your previous dir to mydir

5. run `firebase emulators:start --import=./mydir --export-on-exit ` and now you have imported the data :)

Can anyone speak to if this workaround still works? I’m assuming that you perform these steps on a Mac and ./mydir is data that was created on a Windows machine. Which means that step 1 fails because importing WIndows data on a Mac machine leads to the discussed error.

I can confir that I’m still using the workaround on Elementary OS 6 and Pop!_OS 21.04 when my collegues commit that file from their Windows 10 machines.