nodejs-storage: state.buffer[0].length Cannot read properties of null (reading 'length')

Environment details

OS: Mac 13.5 (22G74)
Node.js version: 21.2.0
npm version: 10.2.3
@google-cloud/storage version: 7.6.0

Getting the following error when trying to download a file:

source-map-support.js:496 node_modules/stream-shift/index.js:16
    return state.buffer[0].length
  source-map-support.js:499 TypeError: Cannot read properties of null (reading 'length')
    at getStateLength (node_modules/stream-shift/index.js:16:28)
    at shift (node_modules/stream-shift/index.js:6:99)
    at Duplexify._forward (node_modules/duplexify/index.js:170:35)
    at PassThrough.onreadable (node_modules/duplexify/index.js:136:10)
    at PassThrough.emit (node:events:519:28)
    at emitReadable_ (node:internal/streams/readable:832:12)
    at processTicksAndRejections (node:internal/process/task_queues:81:21)

This only happens when I download the file from a certain project. If I copy the exact same config and code but initiate it within a script, it works

About this issue

Original URL
State: closed
Created 7 months ago
Reactions: 10
Comments: 19 (2 by maintainers)

Commits related to this issue

fix: 🐛 storage access in node 20 https://github.com/googleapis/nodejs-storage/issues/2368#issuecomment-1914372497 — committed to konkarin/portfolio by konkarin 5 months ago

Most upvoted comments

stream-shift 1.0.2 works fine with node 20.11.0

Can confirm that 💯 we had to add an override to our package.json to enforce stream-shift 1.0.2.

mksony on Jan 25, 2024

I experienced the same issue with Node 20 although my error message was slightly different:

TypeError: Cannot read properties of null (reading 'length')
    at getStateLength (/prod/api/node_modules/.pnpm/stream-shift@1.0.1/node_modules/stream-shift/index.js:16:28)
    at shift (/prod/api/node_modules/.pnpm/stream-shift@1.0.1/node_modules/stream-shift/index.js:6:99)
    at Duplexify._forward (/prod/api/node_modules/.pnpm/duplexify@4.1.2/node_modules/duplexify/index.js:170:35)
    at PassThrough.onreadable (/prod/api/node_modules/.pnpm/duplexify@4.1.2/node_modules/duplexify/index.js:136:10)
    at PassThrough.emit (node:events:518:28)
    at emitReadable_ (node:internal/streams/readable:832:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:81:21)

To work around I changed my code from: await this.storage.bucket(config.gcp.storageBucket).file(gcsPath).download(options);

To: await this.transferManager.downloadFileInChunks(gcsPath, options);

Like in this example downloadFileInChunksWithTransferManager.js

mpalmerlee on Jan 13, 2024

Those who might end up here whilst running a Google Cloud Function, theres a chance the underlying image of your cloud function has been updated and this bug has manifested itself!

For the time being, you can add the following to your package.json to get around the issue the storage API is updated:

JSON

  "overrides": {
    "stream-shift": "1.0.2"
  }

Worth noting that stream-shift is a dependency of duplexify which appears unmaintained…

MattBidewell on Jan 29, 2024

stream-shift 1.0.2 works fine with node 20.11.0

russell-liu-splyt on Jan 16, 2024

For those having problems on Node 21.2 it appears the solution is to downgrade to get around the problem. Looking at the provided stack trace it appears to mainly be coming from Node internal libs / Duplexify. It might be worth opening an issue for the Duplexify folks to see if they can resolve it on their side. We do intend to remove Duplexify from this library in the future (https://github.com/googleapis/nodejs-storage/issues/2041) However, that will not happen until the next major revision to the library which is likely sometime next year.

ddelgrosso1 on Nov 27, 2023

Has to do with this: https://github.com/mafintosh/stream-shift/issues/6

Downgrade your node.js version to below 21.2

gal-checksum on Nov 23, 2023

Can also confirm that setting stream-shift to 1.0.2 for all dependencies (used the overrides method) works with node 20.11.0 🎉

MilosLukicBM on Jan 25, 2024