firebase-tools: Deployment error with no details after updating to Node 8 and firebase-tools@^4.0.0

Hey @thechenky, I did:

Build failed: {“cacheStats”: [{“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “global”}, {“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “project”}]}

I waited about 5 mins as I read this thread online, deployed again and it worked fine.

Is there a reason why the build might still fail at times?

Still experiencing exactly the same issue today. (2020 - 01 - 07)

From Google Support

"We’re currently having an issue with Cloud Functions deployment. Our engineering team is currently working on this to put back the services the soonest possible time. I’ll get back to you with more updates.

Apologies for the inconvenience this may have caused you."

The status page currently says:

We've received a report of an issue with Google Cloud Functions and Google App Engine deployments failing for some customers.

The team is investigating and will post an issue on the dashboard if it’s warranted as they learn more. In the meantime if you’re having a deployment issue with Cloud Functions please write in to Firebase or Google Cloud support! Thank you all for your patience.

Anyone experiencing this error please +1 and weigh in on this from the public Cloud issue tracker: https://issuetracker.google.com/issues/154260223

On Sat, Apr 25, 2020, 4:49 AM chenlevy notifications@github.com wrote:

Same here Deployment error. Anywhere between 1-5 different functions fail to deploy: Build failed: {“cacheStats”: [{“status”: “MISS”, …

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/firebase/firebase-tools/issues/853#issuecomment-619367267, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAH7RZHEUCY2EAR2ABU3DROLE23ANCNFSM4FNSPUCA .

Deleting functions/lib didn’t fix anything for me. Still getting functions[getVerificationCode(us-central1)]: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": …

Error: Functions did not deploy properly.

I can confirm that removing the folder functions/lib fixed the issue for me 😃

Hi all, thank you for these reports. We are aware of this issue and there is ongoing work in the pipeline to mitigate this pain point. I cannot share timelines, but please know we hear you, we know this issue is frustrating, and we’re prioritizing work that will reduce these errors.

Internal bug reference: 117124663

Same here, but I had this working this morning, I think for our case, this is related to the GCF status right now: https://status.firebase.google.com/incident/Functions/18034 @mulhoon @yuliankarapetkov @urkopineda @michaelauderer

I am still getting the same error about HIT and MISS on cache stats. This is happening whole day.

Guys, please report deploy issues here, so they might resolve it finally. @pragati27gupta, @taronaeo, @acoyfellow, @MikeWeiZhou https://firebase.google.com/support/troubleshooter/functions/deploy

Sorry to bring this topic back up. But I do have the same problem.

Everytime I deploy, anywhere from 1 - 5 functions would fail with message like this:

There isn’t a function that consistently fails to deploy, it seems like random functions fail at different times. Maybe 1 in 12 tries I might get all the functions to deploy at once.

Any hints as to any work arounds besides deploying multiple times for failed functions? And I appreciate all the work thats put into this tool. TY

btw. I opened issue at cloudfunctions issue tracker about that: https://issuetracker.google.com/issues/154260223 Please, comment / star the issue to get more traction there.

Build failed: {“cacheStats”: [{“status”: “MISS”, “hash”: “d77e8c491b10d7f6c295d44d41963365d20c4b63b5a98586acce25c13826ff34”, “type”: “docker_layer_cache”, “level”: “global”}, {“status”: “HIT”, “hash”: “d77e8c491b10d7f6c295d44d41963365d20c4b63b5a98586acce25c13826ff34”, “type”: “docker_layer_cache”, “level”: “project”}]}

Facing the same issue today. Was working well till yesterday.

Google just confirmed they are having deployment issues with cloud functions

Hey @thechenky, I did:

Build failed: {“cacheStats”: [{“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “global”}, {“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “project”}]}

I waited about 5 mins as I read this thread online, deployed again and it worked fine.

Is there a reason why the build might still fail at times?

Same here, we have the latest version of firebase tools running on a GitLab runner and this keeps happening. It’s such a big annoyance to keep retrying. Our whole CI/CD pipeline is broken due to this, causing such a waste of developer time, tracking the jobs and clicking retry to eternity.

Worst of it, the Firebase Support team hasn’t been helpful at all and have pointed to some bunch of random docs, totally ignoring the randomness of function deployment errors.

https://status.cloud.google.com/ still pretends everything is up But last I tried, none of my functions deployed 😬

😖

@ahaverty In my case, this is only 3 or 4 functions on 52 that fail. But I’d rather have the deployment fail completely so all our application functions are always on the same version.

Ok interesting, thanks for the additional datapoints. So it seems like it’s not due to the new versions of firebase-tools per se, but due to deploying to Node 8 vs Node 6. (So using the same firebase-tools but deploying to Node 6 will reduce errors). I’ve filed a bug internally with the team responsible for the runtime.

getting the same error and only deploying 12 functions. Never had these deployment issues until upgrading to node 8.

…if there’s a limit to the number you can deploy, shouldn’t it be documented somewhere?

@Trinhlvtq, upgrading firebase-tools won’t solve this issue.

Out of curiosity, how many node dependencies do you use in your project? Do you use native ones in your project?

Those error codes are the exact ones we received previously when Google’s Cloud Function systems had an internal error of themselves.

My best solution for you is either:

1. Keep attempting to deploy until it works (should take not more than 14 retries)
2. Contact Google about the issue and let them check their infrastructure for issues

Same here. Any attempt to deploy even only the default helloWorld firebase function throws the same error: Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "...", "type": "docker_layer_cache", "level": "global"}

Hey @thechenky, I did: Build failed: {“cacheStats”: [{“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “global”}, {“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “project”}]} I waited about 5 mins as I read this thread online, deployed again and it worked fine. Is there a reason why the build might still fail at times?

Still experiencing exactly the same issue today. (2019 - 01 - 07)

*2020-01-07 probably;) And yes I am too experiencing this for some time now…

Same here. But it doesn’t worked after waiting for some time…

Hi @thechenky, has there been any update? This bug is starting to affect our project as other libraries start dropping node v6 support, we’re having to do a lot of workarounds staying on node v6, with this bug being the only blocker from upgrading to v8.

I just switched over to the lastest version of firebase tools, Node 8, and firebase functions last night and unfortunately ran into the same issues mentioned here.

@devth have a look at this message.

@fatihacet As I mentioned earlier, the build errors mostly happen because of a timeout during the deploy process, which has been reduced from 10 to 5 minutes by GCP about 2 weeks ago.

We also had a retry system in place, implemented in a fork of firebase-tools (and usable) here (PR for official firebase-tools is here #1977).

However, we had been stuck with our deployments for more than 2 weeks due to the recent timeout decrease, and retries wouldn’t solve the problem for us, as we couldn’t deploy a single function on any of our environments, even with 3 to 5 retries, during this period.

The only fix is to reduce the amount of dependencies that you are using for your cloud functions. In our case, we had to compile them separately and bundle all that could be in the function’s code using webpack.

With this update on the build system timeout, the firebase model of using a single entry point for all of the functions to be deployed simply doesn’t work anymore.

To anyone that still encounter deployment errors:

After first fixing our deployments by switching on Node 10 and running with the latest version of firebase-tools, we recently (about 2 weeks ago) encountered errors again with an opaque Build error: details not available. We were completely stuck because this happened constantly for all of our functions on all our environments.

After investigating, we noticed that the build time for cloud functions had been reduced on GCP from 10 minutes to only 5 minutes.

To improve build time, GCP uses caching to re-install node_modules based on the hash of either the package-lock.json or the yarn.lock files, depending on which you are using. But if your dependencies are updated and this file changes, GCP has to re-install all of them, which was taking too long in our case, especially since we have many native modules that need to be rebuilt. I think this is what we can see in the Node 8 error details with the "status": "MISS".

We ended up building our own tooling to compile our functions separately, bundling all non-native modules in our main function code using webpack, and having a trimmed package.json file for each function with only the missing dependencies for each of them.

Now our deployments work well every time, and take 1 to 2 minutes max per function.

So, firebase uses docker. That’s fun to know.

Thanks for reaching out! Your request has been submitted and we’ll reach out to you soon.

@selvinfehric has the right idea.

https://status.cloud.google.com/incident/cloud-functions/20001

Is it safe to delete functions/lib ? Does it get regenerated every time with firebase deploy?

It is completely safe to delete that folder. Whenever you run firebase deploy, it will automatically create that folder with the contents inside.

Same here. Also having the same problem:

!  functions[createUserAccount(us-central1)]: Deployment error.
Build failed: {"cacheStats": [{"status": "MISS", "hash": "e03d8d6f6bf22fcb1cf50f4e12b3e3a3b59954fade707db62e79562c9dbac3ef", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "e03d8d6f6bf22fcb1cf50f4e12b3e3a3b59954fade707db62e79562c9dbac3ef", "type": "docker_layer_cache", "level": "project"}]}


Functions deploy had errors with the following functions:
        createUserAccount

with the following code:

await database.collection('members').where('Email', '==', user.email).limit(1).get()
    .then(snapshot => {
      snapshot.forEach(doc => {
        console.log(doc.id, '=>', doc.data())
      })
    }).catch(err => console.error(err))

I have just checked the google services on the following link that they have claimed all are up. https://status.cloud.google.com/ We have to spend three hours for issue investigation. Please update the confirm status with some URL here Thanks,

They’ve updated the page now

Hi,

I am also getting a similar error

Deployment error. Build failed: {"cacheStats": [{"status": "MISS", "hash": "{hashCode}", "type": "docker_layer_cache", "level": "global"}, {"status": "HIT", "hash": "{hashCode}", "type": "docker_layer_cache", "level": "project"}]}

Yep, just experienced this, and landed here from search results

Build failed: {"cacheStats": [{"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "{{hashCode}}", "type": "docker_layer_cache", "level": "project"}]}

I think it’s a different issue?

Hey @thechenky, I did: Build failed: {“cacheStats”: [{“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “global”}, {“status”: “MISS”, “hash”: “{{hashCode}}”, “type”: “docker_layer_cache”, “level”: “project”}]} I waited about 5 mins as I read this thread online, deployed again and it worked fine. Is there a reason why the build might still fail at times?

Still experiencing exactly the same issue today. (2019 - 01 - 07)

Yep I have the same issue as well

We have been using node 8 on 32 functions since it was available and only started having this issue last night. Deploying fails for between 1 and 5 functions, and they seem to be different every time.

@jojonarte the only resolution I found was to refactor back to Node 6, deploy an empty index.js to delete all functions (otherwise they remain Node 8 functions), and then deploy the Node 6 index.js

@jojonarte There is no quick fix currently, the current massive errors are due to the Firebase platform encountering a service disruption as stated by @gautier-gdx. However, we still have problems deploying all our functions at once with Node 8 when Firebase is up.