cli: 502 Bad Gateway when deploying large site

- Do you want to request a feature or report a bug?

Bug, sort of. We might be stretching what’s classed as supported behaviour, in which case I’m requesting that that scope be widened to sites of this size.

- What is the current behavior?

When attempting to deploy a large site (upwards of 20000, sometimes 50000 total files) via the netlify CLI, we often see the following error either during the CDN diffing files phase or (as in this case) the Uploading files phase:

Deploying to live site URL...
- Hashing files...
✔ Finished hashing 57892 files
- CDN diffing files...
✔ CDN requesting 29854 files
- Uploading 29854 files
 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at NetlifyAPI.uploadDeployFile (/usr/local/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)
    at <anonymous>
Exited with code 2

This is from a CI server environment, so one with a (presumably) pretty reliable and fast internet connection.

When run on a local machine, we get slightly more overview of the behaviour. The upload appears to work as expected right up until the end:

Deploying to live site URL...
✔ Finished hashing 57893 files
✔ CDN requesting 30907 files
⠋ (30906/30907) Uploading s/404.html...
 ›   Warning: TextHTTPError: 502 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
⠒ (30906/30907) Uploading s/404.html...TextHTTPError: Bad Gateway
    at NetlifyAPI.uploadDeployFile (~/.nvm/versions/node/v8.11.4/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)
    at <anonymous>

You can see above, the tool claims the bad gateway on the very last file (which in this case happens to be 404.html). Re-running the deploy at this point usually results in the CDN saying it doesn’t need any of the files uploading again, and successfully finishing.

- If the current behavior is a bug, please provide the steps to reproduce.

I think reproduction steps might be to generate a site with around 50000 files and try to deploy it. I can try to form a simple Gatsby setup that might generate 50000 unique-looking files on each build if needed.

- What is the expected behavior?

A successful deploy.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 17 (2 by maintainers)

Commits related to this issue

Increase netlify deploy timeout to 10min As suggested in netlify/cli#484. — committed to owid/owid-grapher by danielgavrilov 4 years ago

Most upvoted comments

I have the same issue for a very large site :

✔ Finished hashing 810170 files
- CDN diffing files...

 ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›
TextHTTPError: Bad Gateway
    at NetlifyAPI.createSiteDeploy (~/.nvm/versions/node/v11.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/open-api/index.js:173:13)

jtraulle on Aug 30, 2019

A question! Does npm publish command also counts the hidden directories (.) and the node_modules files?

Hi @actuallyakash, do you mean netlify deploy? The logic for that is here: https://github.com/netlify/cli/blob/2b53f4a707f7d86ae97154a7b6dc438ac8bca5ab/src/commands/deploy.js#L146

You can read about npm publish here.

If you have any other questions I suggest opening a new issue.

erezrokah on Jan 11, 2021

I’m on the low end of site size here (~4800 pages), and noticed this just started happening:

$ netlify deploy
Deploy path: ~/code/blog/public
Deploying to draft URL...
✔ Finished hashing 4799 files
⊷ CDN diffing files... ›   Warning: TextHTTPError: 502
 ›   Warning: 
 ›   TextHTTPError: Bad Gateway
 ›

I got around it eventually by manually setting a 10 minute timeout: netlify deploy --timeout 6000

$ netlify deploy --timeout 6000 --message "with a timeout"
Deploy path: ~/code/blog/public
Deploying to draft URL...
✔ Finished hashing 4799 files✔ CDN requesting 2417 files
✔ Finished uploading 2417 assets✔ Draft deploy is live!

Not an ideal solution… but hopefully it helps others! /shrug

stephencweiss on May 29, 2020