firebase-tools: Large, below-quota deployments (<80 functions) failing with message "Build failed with status: EXPIRED..."

[REQUIRED] Environment info

firebase-tools: 12.3.1

Platform: macOS Monterey

[REQUIRED] Test case

Run firebase deploy --only functions with a codebase containing 61 2nd Cloud Functions and 1 1st gen Cloud Function.

[REQUIRED] Steps to reproduce

Same as test case

[REQUIRED] Expected behavior

I would expect this deployment to go through since it is below the deployment quota of 80 functions per minute.

[REQUIRED] Actual behavior

I saw that firebase-tools had a recent update to improve retry behavior for large deployments so I thought I would try it. I am still running into issues deploying under 80 (quota) functions, however. It seems like the deployment times out before all the function deployments go through.

Some of the functions fail to deploy with build with the following error:

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].

The build logs are empty and don’t contain any useful information other than the build expiring.

This happens for about 20 of the 62 functions. Here is a truncated output of the full deploy:

=== Deploying to 'project'...

i  deploying functions
Running command: 
✔ Preparing the list of your Firebase projects

> lint
> eslint .


✔  functions: Finished running predeploy script.
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: Loaded environment variables from .env, .env.staging.
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (hidden MB) for uploading
i  functions: packaged functions (hidden MB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API eventarc.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (1st Gen) function dailyBackup(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
i  functions: updating Node.js 16 (2nd Gen) function functionName(us-central1)...
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted]
✔  functions[functionName(us-central1)] Successful update operation.
✔  functions[functionName(us-central1)] Successful update operation.

Functions deploy had errors with the following functions:
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
	functionName(us-central1)
Function URL (functionName(us-central1)): redacted
Function URL (functionName(us-central1)): redacted
Function URL (functionName(us-central1)): redacted
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1
- Error Failed to update function functionName in region us-central1

This is a small portion of the --debug log that printed for each failed function, but most of it did not seem useful:

  "original": {
    "name": "FirebaseError",
    "children": [],
    "exit": 1,
    "message": "Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted],
    "status": 3,
    "code": 3
  }

About this issue

Original URL
State: closed
Created a year ago
Reactions: 9
Comments: 19 (2 by maintainers)

Most upvoted comments

firebase-tools >=v12.6.1 includes the enabling of single builds for v2 functions, which should drastically improve deployment speed / reliability on large deployments. The feature should ameliorate the build expiring problem mentioned in this issue by reusing builds from the same deploy group. Will go ahead and close this thread for now, but please open a new issue if you encounter any other errors while deploying functions using the latest versions of the CLI tool.

blidd-google on Oct 11, 2023

We see this a lot too. It’s very frustrating.

lox on Aug 15, 2023

@gillycheesesteak We’re still working on ironing out some minor bugs with it, but https://github.com/firebase/firebase-tools/pull/6376 should lead to some major improvements to this experience!

joehan on Oct 3, 2023

Any news on this? @Berlioz @colerogers sorry to tag you 🥺

gillycheesesteak on Oct 3, 2023

Also been dealing with this issue. Ever since I updated my 30 or so cloud functions to v2, I’ve been noticing errors when deploying some functions (the problematic ones change every time):

Build failed with status: EXPIRED and message: An unexpected error occurred. Refer to build logs: [redacted]. For more details see the logs at [redacted].
✔  functions[payment-deleteOldPayments(us-central1)] Successful update operation.
✔  functions[customer-deleteJobsOnCustomerDelete(us-central1)] Successful update operation.
✔  functions[hourEntry-sendHourEntryReminders(us-central1)] Successful update operation.
✔  functions[calendar-addEvent(us-central1)] Successful update operation.
Functions deploy had errors with the following functions:
        calendar-getEvents(us-central1)
        job-deletePaymentsOnJobDelete(us-central1)
        payment-deletePaymentCheckPhoto(us-central1)
        subcontractor-onSubcontractorDelete(us-central1)
        user-deleteTasksOnUserDelete(us-central1)
Function URL (user-registerDeviceToken(us-central1)): [redacted]
Function URL (user-createNewUser(us-central1)): [redacted]
Function URL (calendar-getEventsToday(us-central1)): [redacted]
Function URL (calendar-addEvent(us-central1)): [redacted]
Function URL (calendar-updateEvent(us-central1)): [redacted]
Function URL (calendar-deleteEvent(us-central1)): [redacted]
i  functions: cleaning up build files...
Error: There was an error deploying functions:
- Error Failed to update function calendar-getEvents in region us-central1
- Error Failed to update function payment-deletePaymentCheckPhoto in region us-central1
- Error Failed to update function user-deleteTasksOnUserDelete in region us-central1
- Error Failed to update function subcontractor-onSubcontractorDelete in region us-central1
- Error Failed to update function job-deletePaymentsOnJobDelete in region us-central1
Warning: run-commands command "firebase deploy --config firebase.json" exited with non-zero status code

Even worse, when these functions fail to deploy, subsequent deploys tell me that there are “no changes detected”. There definitely are since those changes never made it to my function to begin with:

=== Deploying to 'crown-finish-development'...
i  deploying storage, firestore, functions, hosting
i  firebase.storage: checking apps/back-end/storage.rules for compilation errors...
✔  firebase.storage: rules file apps/back-end/storage.rules compiled successfully
i  firestore: reading indexes from apps/back-end/firestore.indexes.json...
i  cloud.firestore: checking apps/back-end/firestore.rules for compilation errors...
✔  cloud.firestore: rules file apps/back-end/firestore.rules compiled successfully
i  functions: preparing codebase default for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  functions: required API cloudbuild.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
i  functions: Loading and analyzing source code for codebase default to determine what to deploy
Serving at port 8191
shutdown requested via /__/quitquitquit
i  functions: preparing dist/apps/back-end directory for uploading...
i  functions: packaged /Users/tylerhendrickson/Developer/soluna-studios/crown-finish/dist/apps/back-end (91.11 KB) for uploading
i  functions: ensuring required API cloudscheduler.googleapis.com is enabled...
✔  functions: required API cloudscheduler.googleapis.com is enabled
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API storage.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API eventarc.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
i  storage: latest version of apps/back-end/storage.rules already up to date, skipping upload...
i  firestore: deploying indexes...
✔  firestore: deployed indexes in apps/back-end/firestore.indexes.json successfully for (default) database
i  firestore: latest version of apps/back-end/firestore.rules already up to date, skipping upload...
i  hosting[crown-finish-development]: beginning deploy...
i  hosting[crown-finish-development]: found 50 files in dist/apps/web
i  hosting: upload complete
✔  hosting[crown-finish-development]: file upload complete
✔  storage: released rules apps/back-end/storage.rules to firebase.storage
✔  firestore: released rules apps/back-end/firestore.rules to cloud.firestore
i  functions: Skipping the deploy of unchanged functions with experimental support for skipdeployingnoopfunctions
✔  functions[user-deleteTasksOnUserDelete(us-central1)] Skipped (No changes detected)
✔  functions[user-deleteHourEntriesOnUserDelete(us-central1)] Skipped (No changes detected)
✔  functions[user-registerDeviceToken(us-central1)] Skipped (No changes detected)
✔  functions[user-createNewUser(us-central1)] Skipped (No changes detected)
✔  functions[job-deletePaymentsOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteReceiptsOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteJobPhotosOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteHourEntriesOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[job-deleteTasksOnJobDelete(us-central1)] Skipped (No changes detected)
✔  functions[hourEntry-sendHourEntryReminders(us-central1)] Skipped (No changes detected)
✔  functions[task-sendNotificationOnAssignment(us-central1)] Skipped (No changes detected)
✔  functions[task-sendTaskReminders(us-central1)] Skipped (No changes detected)
✔  functions[payment-deletePaymentCheckPhoto(us-central1)] Skipped (No changes detected)
✔  functions[payment-deleteOldPayments(us-central1)] Skipped (No changes detected)
✔  functions[receipt-deleteReceipt(us-central1)] Skipped (No changes detected)
✔  functions[receipt-deleteOldReceipts(us-central1)] Skipped (No changes detected)
✔  functions[estimate-deleteJobPhotosOnEstimateDelete(us-central1)] Skipped (No changes detected)
✔  functions[estimate-sendEstimateReminders(us-central1)] Skipped (No changes detected)
✔  functions[jobPhoto-deleteJobPhoto(us-central1)] Skipped (No changes detected)
✔  functions[jobPhoto-deleteOldJobPhotos(us-central1)] Skipped (No changes detected)
✔  functions[customer-updateJobsOnCustomerRename(us-central1)] Skipped (No changes detected)
✔  functions[customer-updateEstimatesOnCustomerRename(us-central1)] Skipped (No changes detected)
✔  functions[customer-deleteJobsOnCustomerDelete(us-central1)] Skipped (No changes detected)
✔  functions[customer-deleteEstimatesOnCustomerDelete(us-central1)] Skipped (No changes detected)
✔  functions[subcontractor-onSubcontractorDelete(us-central1)] Skipped (No changes detected)
✔  functions[calendar-getEvents(us-central1)] Skipped (No changes detected)
✔  functions[calendar-getEventsToday(us-central1)] Skipped (No changes detected)
✔  functions[calendar-addEvent(us-central1)] Skipped (No changes detected)
✔  functions[calendar-updateEvent(us-central1)] Skipped (No changes detected)
✔  functions[calendar-deleteEvent(us-central1)] Skipped (No changes detected)
✔  functions[expiration-recordCreationTime(us-central1)] Skipped (No changes detected)
i  functions: cleaning up build files...
i  hosting[crown-finish-development]: finalizing version...
✔  hosting[crown-finish-development]: version finalized
i  hosting[crown-finish-development]: releasing new version...
✔  hosting[crown-finish-development]: release complete
✔  Deploy complete!

Now I can’t trust my CI/CD to deploy reliably. Some changes make it while others don’t until I deploy next time with new changes. This is a huge problem. Can we please have some movement on this?

hendrickson-tyler on Aug 9, 2023