firebase-tools: Storage hangs without completing or errors when uploading or deleting files in parallel

[REQUIRED] Environment info

firebase-tools: 11.7.0

Platform: Windows 10 Pro running Ubuntu via WSL2

[REQUIRED] Test case

As you start to increase the number of items attempted to be uploaded to the storage emulator you will increase the chances of that upload failing. It will fail will even relatively small numbers of parallel uploads if attempted enough time (e.g. 2 or 3 files). I believe I also had it fail with a single file but haven’t been able to reproduce as the failure is much less likely.

For example, if you try to upload 5x files via an approach similar to the one below: The first time maybe 3/5 will upload next 2/5 next 4/5 next 5/5

[REQUIRED] Steps to reproduce

`

// Upload Issue

const uploadFiles = () => {
  let uploadTasks = [];
  for (let i = 0; i < files.value.length; i++) {
    let curFile = files.value[i];

    let storageRef = fb_ref(storage, "imgs/" + curFile.name);
    const uTask = uploadBytesResumable(storageRef, curFile);
    uploadTasks.push(uTask);
    // Listen for state changes, errors, and completion of the upload.
    uploadTasks[i].on(
      "state_changed",
      (snapshot) => {
        // Get task progress, including the number of bytes uploaded and the total number of bytes to be uploaded

        console.log("stage changed for ", i, snapshot);
        const progress = snapshot.bytesTransferred / snapshot.totalBytes;

        //Update Progress
        uploadProgress.value[i].percent = progress;
        uploadProgress.value[i].color = "green-2";
        if (i == 1) {
          uploadProgress.value[i].color = "purple-2";
        }

        const progressPercent = progress * 100;
        // console.log("Upload is " + progressPercent + "% done");
        switch (snapshot.state) {
          case "paused":
            console.log("Upload is paused");
            break;
          case "running":
            console.log("Upload is running");
            break;
        }
      },
      (error) => {
        console.log("failed ============================== ", error);
        switch (error.code) {
          case "storage/unknown":
            // Unknown error occurred, inspect error.serverResponse
            break;
        }
      },
      () => {
        // Upload completed successfully, now we can get the download URL
        //Update Progress
        console.log("Trying to get url for foo: ", i);
        getDownloadURL(uploadTasks[i].snapshot.ref)
          .then((downloadURL) => {
            console.log("File Download foo: " + i, downloadURL);
          })
          .catch((e) => {
            console.log("error foo: ", e);
          });
      }
    );
  }
  Promise.all(uploadTasks).then(() => {
    console.log("Files done uploading");
  });`

// Delete Issue
  try {
    let deletePromises = [];
    console.log("deleting storage ", imgArray);
    imgArray.forEach((img) => {
      let imgPath = `${img.path}${img.name}`;
      let deleteRef = ref(storage, imgPath);
      console.log("path: ", imgPath);
      deletePromises.push(deleteObject(deleteRef));
    });
    await Promise.all(deletePromises);
  } catch (e) {
    console.log("Error: ", e);
  } finally {
    Loading.hide();
  }

[REQUIRED] Expected behavior

It should be possible to consistently upload or delete multiple files in parallel to storage

[REQUIRED] Actual behavior

The process will randomly hang without showing any errors. It just never completes.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 24 (10 by maintainers)

Most upvoted comments

I’m seeing the same issue. I get lines like this in the console and the storage emulator stops responding.

{"result":{"permit":false},"id":18,"status":"ok"}
{"result":{"permit":false},"id":19,"status":"ok"}

I’m calling getDownloadUrl to fetch multiple images from storage at once. It doesn’t break every time, but it’s often enough that the storage emulator is basically useless.

I’m seeing a similar issue only with getDownloadUrl() Using the emulators and looping through image references, it hangs on the promise.

const imagesMapped = []
  for (const imagePath of arrayOfImagesPath) {
    const storageRef = ref(storage, imagePath)
    try {
      const storageUrl = await getDownloadURL(storageRef)
      imagesMapped.push(storageUrl)
    } catch (e) {
      console.error(e)
      throw e
    }
  }

Doesn’t appear to be a staging or production issue

I experienced the same issue on MacOS with firebase-tools 11.8.0