firebase-tools: V2 Functions crashing when deployed with 12.6.1

Hi all, we’ve been able to reproduce the error and have identified the underlying bug. We’re currently working on a fix and will keep folks updated on this issue thread. Thanks for your continued patience as we iron out the remaining kinks with the single builds feature - we believe that the improvements in deploy speed & reliability that single builds offer are worth the effort, and you should be able to take advantage of those improvements without problems very soon!

@blidd-google are you able to address these questions:

1. Did this affect all V2 functions? What was the identified issue?
2. If functions were being incorrectly invoked, could there have been security impact? For instance onDocumentUpdate functions triggered by http without auth checks?
3. Why was a change of this magnitude introduced in a minor version? Will the release regime in future change?

My confidence in Firebase Functions is at an all time low after this issue and the recent multi-hour deployment outage that didn’t even merit a status page update.

Would really help to see a nuanced response here, are there any product managers involved in Firebase any more or is it more best-effort internally?

I want to add: I experienced the same issue with onDocumentWrite & onSchedule with v2 functions.

• Downgrade to firebase-tools: 12.5.2 did work.
• Deploying an individual function which is onSchedule does also work.
• Deploying a set of functions is the one case where it breaks for all of these functions.
• The functions that keep working are the http request functions.

It’s crazy that this bug is not fixed at this time! I second @lox in saying that this is critical and more importantly: a serious break in trust for future updates (which are part of a minor version bump and multiple releases at this point)!

First off, a quick update: We have a backend fix for this issue rolling out to production. It should land this week.

To answer @lox 's questions:

Did this affect all V2 functions? What was the identified issue? Though there were a couple issues related to this launch, this specific issue only affected deployments with multiple V2 functions using different trigger types.

A brief bit of background: previously, deployments of many functions would generate some nearly identical artifacts for each function. This update used ‘source tokens’ to tell Cloud Functions to reuse these artifacts where possible to make deploys faster, cheaper, and more reliable.

The cause of this issue is a bug where functions deployed with a source token used the trigger type of the function that created that source token instead of the correct trigger type. This trigger type controls how the Cloud Function decodes requests and responses. So, for example, if a source token from a Firestore triggered function was used for a HTTPS function, it would error out when trying to decode a HTTPS request as a request from Firestore.

If functions were being incorrectly invoked, could there have been security impact? For instance onDocumentUpdate functions triggered by http without auth checks? This issue did not have any security impact. The correct functions are being called, this was just causing errors with how requests were decoded.

Why was a change of this magnitude introduced in a minor version? Will the release regime in future change? We added this in a minor version because it is not an ‘incompatible API change’, per https://semver.org/.

We do understand the frustration here, and we definitely need to do a better job of catching issues like this before they reach you. This issue exposed some gaps in our test coverage, and we’re currently working on a revamp the functions integration test suite. Additionally, while Github is a great place to report issues, please reach out to Firebase Support for help with any billing issues.

I’m going to close this issue for now since AFAIK, it is fixed. If you’ve had any issues with your functions, the quickest way to fix them should be to update to the latest version of firebase-tools and redeploy them. Please don’t hesitate to ping here again (or open new issues) if you are still having trouble with functions deploy.

The backend fix for the single builds feature has been fully released; functions deployed with CLI versions >=12.6.2 will be able to correctly decode requests. We will continue to monitor this thread in case anything else comes up, but thank you all for your patience as we ironed out the kinks for this feature. As @joehan mentioned, the Firebase functions team is working on overhauling our integration tests to mitigate potential future issues as we continue to make improvements to 2nd gen functions.

@ChromeQ I am experiencing the issue myself, not a Firebase dev 😅

I also got problems at runtime after deploying onCall v2 functions with with 12.6.1. TypeError: res.on is not a function

@joehan has the backend fix landed in production? Thanks!

Thanks for the detailed update @joehan!

Very much looking forward to being able to benefit from artifact re-use (and speed up our deploys).

Yup, absolutely. I am also happy to do a video call at a time that suits showcasing the issue.

Same on my end, downgrade to 12.5.2 and everything works again. Got a mix of v2 and v1 (needed for firebase auth triggers). Deployment on 12.6.1 is extremely fast, over 10x faster than 12.5.2, which makes me think that something is not right. In my case, onRequest is not an issue, the onSchedule functions trigger the same issue “TypeError: Cannot read properties of undefined (reading ‘headers’)”.

We have a support case 10253449 already open

Hey @milo-, sorry to hear you’re encountering this issue. I’m currently trying to reproduce the error you mentioned but so far I am unable to replicate this. I tried deploying a mixture of v1 and v2 HTTP functions:

const functionv1 = require("firebase-functions/v1");
const functionv2 = require("firebase-functions/v2");
const logger = require("firebase-functions/logger");

exports.helloWorld1 = functionv1.https.onRequest((request, response) => {
  const rawBody = request.rawBody.toString();
  logger.info(`rawBody: ${rawBody}`, { structuredData: true });
  response.send(`Hello from Firebase v1! ${rawBody}`);
});

exports.helloWorld2 = functionv2.https.onRequest((request, response) => {
  const rawBody = request.rawBody.toString();
  logger.info(`rawBody: ${rawBody}`, { structuredData: true });
  response.send(`Hello from Firebase v2 2! ${rawBody}`);
});

exports.helloWorld3 = functionv2.https.onRequest((request, response) => {
  const rawBody = request.rawBody.toString();
  logger.info(`rawBody: ${rawBody}`, { structuredData: true });
  response.send(`Hello from Firebase v2 3! ${rawBody}`);
});

Making a POST request to any of the HTTP functions above does not raise any errors. In case I’m missing anything here, please let me know.

Could you provide additional information here such as if you’re deploying a mixture of v1 and v2 functions or just multiple v2 functions? By any chance, if you can provide a sample code snippet of the function(s) you’re deploying, we would highly appreciate it.