node-gyp: Random HTTP 500 statuses or ECONNRESET when node-gyp fetches dependencies

This is occurring across multiple node versions, architectures and system types. Our build system regularly runs node gyp in its build for our NPM packages across every version of nodejs and about 5% of builds are subject to a 500 status failure or an ECONNRESET when setting up node-gyp.

This strikes me as an issue with the server which hosts the node gyp files. Is there possibly a way around downloading these node-gyp dependencies each time? Or maybe something that can be done to fix the instability there?

These builds all took place inside of Amazon server instances.

Here is an instance on node 14 with node gyp 7.1.2 on a Windows Server 2019 machine.

04:08:09  gyp info ok 
04:08:13  Now using node v14.21.2 (64-bit)
04:08:19  gyp info it worked if it ends with ok
04:08:19  gyp info using node-gyp@7.1.2
04:08:19  gyp info using node@14.21.2 | win32 | x64
04:08:19  gyp info find Python using Python version 3.8.5 found at "C:\Python38\python3.exe"
04:08:23  gyp http GET https://nodejs.org/download/release/v14.21.2/node-v14.21.2-headers.tar.gz
04:08:23  gyp http 200 https://nodejs.org/download/release/v14.21.2/node-v14.21.2-headers.tar.gz
04:08:25  gyp http GET https://nodejs.org/download/release/v14.21.2/SHASUMS256.txt
04:08:25  gyp http GET https://nodejs.org/download/release/v14.21.2/win-x86/node.lib
04:08:25  gyp http GET https://nodejs.org/download/release/v14.21.2/win-x64/node.lib
04:08:25  gyp http GET https://nodejs.org/download/release/v14.21.2/win-arm64/node.lib
04:08:25  gyp http 200 https://nodejs.org/download/release/v14.21.2/SHASUMS256.txt
04:08:26  gyp http 500 https://nodejs.org/download/release/v14.21.2/win-arm64/node.lib
04:08:26  gyp WARN install got an error, rolling back install
04:08:26  gyp ERR! configure error 
04:08:26  gyp ERR! stack Error: 500 status code downloading arm64 node.lib
04:08:26  gyp ERR! stack     at Request.<anonymous> (C:\nvm\v14.21.2\node_modules\node-gyp\lib\install.js:335:22)
04:08:26  gyp ERR! stack     at Request.emit (events.js:412:35)
04:08:26  gyp ERR! stack     at Request.onRequestResponse (C:\nvm\v14.21.2\node_modules\node-gyp\node_modules\request\request.js:1059:10)
04:08:26  gyp ERR! stack     at ClientRequest.emit (events.js:400:28)
04:08:26  gyp ERR! stack     at HTTPParser.parserOnIncomingClient (_http_client.js:647:27)
04:08:26  gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:127:17)
04:08:26  gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:515:22)
04:08:26  gyp ERR! stack     at TLSSocket.emit (events.js:400:28)
04:08:26  gyp ERR! stack     at addChunk (internal/streams/readable.js:293:12)
04:08:26  gyp ERR! stack     at readableAddChunk (internal/streams/readable.js:267:9)
04:08:26  gyp ERR! System Windows_NT 10.0.17763
04:08:26  gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild" "--msvs_version=2017" "--devdir=C:\\tmp\\.gyp"

Here is an instance of it occurring on Linux ARM64 with electron & node 16 and node-gyp 9.2.0.

02:06:10  gyp info it worked if it ends with ok
02:06:10  gyp info using node-gyp@9.2.0
02:06:10  gyp info using node@16.19.0 | linux | arm64
02:06:10  gyp info find Python using Python version 3.6.8 found at "/usr/bin/python3"
02:06:10  gyp http GET https://electronjs.org/headers/v9.0.0/node-v9.0.0-headers.tar.gz
02:06:11  gyp WARN install got an error, rolling back install
02:06:11  gyp ERR! configure error 
02:06:11  gyp ERR! stack FetchError: request to https://www.electronjs.org/headers/v9.0.0/node-v9.0.0-headers.tar.gz failed, reason: read ECONNRESET
02:06:11  gyp ERR! stack     at ClientRequest.<anonymous> (/home/pdftron/.nvm/versions/node/v16.19.0/lib/node_modules/node-gyp/node_modules/minipass-fetch/lib/index.js:130:14)
02:06:11  gyp ERR! stack     at ClientRequest.emit (node:events:513:28)
02:06:11  gyp ERR! stack     at TLSSocket.socketErrorListener (node:_http_client:494:9)
02:06:11  gyp ERR! stack     at TLSSocket.emit (node:events:525:35)
02:06:11  gyp ERR! stack     at emitErrorNT (node:internal/streams/destroy:157:8)
02:06:11  gyp ERR! stack     at emitErrorCloseNT (node:internal/streams/destroy:122:3)
02:06:11  gyp ERR! stack     at processTicksAndRejections (node:internal/process/task_queues:83:21)
02:06:11  gyp ERR! System Linux 5.10.0-12-cloud-arm64
02:06:11  gyp ERR! command "/home/pdftron/.nvm/versions/node/v16.19.0/bin/node" "/home/pdftron/.nvm/versions/node/v16.19.0/bin/node-gyp" "rebuild" "--target=v9.0.0" "--dist-url=https://electronjs.org/headers" "--openssl_fips=" "--arch=arm"
02:06:11  gyp ERR! cwd /home/admin/jenkins/workspace/Node_Linux_Electron_Arm64_master/WebPDFWorker/Node.js/Packaging
02:06:11  gyp ERR! node -v v16.19.0
02:06:11  gyp ERR! node-gyp -v v9.2.0
02:06:11  gyp ERR! not ok 

And on Linux x64 with node 9, node-gyp 5.1.1

01:50:03  Now using node v9.11.2 (npm v5.6.0)
01:50:03  gyp info it worked if it ends with ok
01:50:03  gyp info using node-gyp@5.1.1
01:50:03  gyp info using node@9.11.2 | linux | x64
01:50:03  gyp info find Python using Python version 2.7.5 found at "/usr/bin/python"
01:50:03  gyp http GET https://nodejs.org/download/release/v9.11.2/node-v9.11.2-headers.tar.gz
01:50:03  gyp http 500 https://nodejs.org/download/release/v9.11.2/node-v9.11.2-headers.tar.gz
01:50:03  gyp WARN install got an error, rolling back install
01:50:03  gyp ERR! configure error 
01:50:03  gyp ERR! stack Error: 500 response downloading https://nodejs.org/download/release/v9.11.2/node-v9.11.2-headers.tar.gz
01:50:03  gyp ERR! stack     at Request.<anonymous> (/home/pdftron/.nvm/versions/node/v9.11.2/lib/node_modules/node-gyp/lib/install.js:188:14)
01:50:03  gyp ERR! stack     at Request.emit (events.js:185:15)
01:50:03  gyp ERR! stack     at Request.onRequestResponse (/home/pdftron/.nvm/versions/node/v9.11.2/lib/node_modules/node-gyp/node_modules/request/request.js:1059:10)
01:50:03  gyp ERR! stack     at ClientRequest.emit (events.js:180:13)
01:50:03  gyp ERR! stack     at HTTPParser.parserOnIncomingClient (_http_client.js:540:21)
01:50:03  gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:117:17)
01:50:03  gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:445:20)
01:50:03  gyp ERR! stack     at TLSSocket.emit (events.js:180:13)
01:50:03  gyp ERR! stack     at addChunk (_stream_readable.js:274:12)
01:50:03  gyp ERR! stack     at readableAddChunk (_stream_readable.js:261:11)
01:50:03  gyp ERR! System Linux 5.10.0-19-cloud-amd64
01:50:03  gyp ERR! command "/home/pdftron/.nvm/versions/node/v9.11.2/bin/node" "/home/pdftron/.nvm/versions/node/v9.11.2/bin/node-gyp" "rebuild"
01:50:03  gyp ERR! cwd /home/admin/jenkins/workspace/PDFNetNode_Linux_master/WebPDFWorker/Node.js/Packaging
01:50:03  gyp ERR! node -v v9.11.2
01:50:03  gyp ERR! node-gyp -v v5.1.1
01:50:03  gyp ERR! not ok

These were just the instances I was able to find within the last 2 weeks of our daily build system. If you want more specific details about each machine I can provide those. The Linux builds are all run within Docker containers.