cli-microsoft365: Bug report: Large file downloading from SharePoint raise RangeError

As soon as @appieschot submits a PR, we’ll try to get it in as quickly as possible.

A little more detail; using m365 spo file get --webUrl 'https://tenant.sharepoint.com/sites/crisismanagement' --url '/sites/crisismanagement/Docs2/Test/5GB.db' --asFile --path 'C:/temp/5GB.db'

I had the following experience:

• the node process downloads with about 66mb/s and grows accordingly to around 5.2gb (Given the 8gb limit I started killing of some other processes like teams)
• At 5.2GB memory consumption it started writing to disk
• It did fail with the following error:

internal/validators.js:85
      throw new ERR_OUT_OF_RANGE(name, `>= ${min} && <= ${max}`, value);
      ^

RangeError [ERR_OUT_OF_RANGE]: The value of "length" is out of range. It must be >= 0 && <= 2147483647. Received 5_368_709_120
    at Function.concat (buffer.js:540:5)
    at Request.<anonymous> (C:\Git\pnp\office365-cli\node_modules\request\request.js:1133:30)
    at Request.emit (events.js:223:5)
    at IncomingMessage.<anonymous> (C:\Git\pnp\office365-cli\node_modules\request\request.js:1083:12)
    at Object.onceWrapper (events.js:312:28)
    at IncomingMessage.emit (events.js:228:7)
    at endReadableNT (_stream_readable.js:1185:12)
    at processTicksAndRejections (internal/process/task_queues.js:81:21) {
  code: 'ERR_OUT_OF_RANGE'
}

Going through the code it of the request.js file it seems to fail when putting the repsone from the buffer to the body that it returns. The only thing I can come up with is to use a filestream instead of the body response using the pipe option request('http://google.com/doodle.png').pipe(fs.createWriteStream('doodle.png')) but I am not sure if that would work.

@waldekmastykarz Yes I do have some test scenario’s for the move issue we are having. Tested with 10mb, 25mb, 100mb, 1gb. Those seem to download perfectly with version 3.0.0 of the CLI on windows. But after reading the issue it might has to do with the fs.readfilesync which has a 2GB buffer limit in node x64. I will try and find a 5gb file to test it with later today.

@tobyatgithub we have just merged the fix for this issue and it’s available in the latest beta version v3.2. You can try it by installing the beta using: npm i -g @pnp/cli-microsoft365@next. Once again, thank you for reporting! 👏

Minor update; got all code working. Need to update two more tests to get it working properly. Sorry for the delay, but end of this week still sounds viable 😉.

@tobyatgithub it looks like I have managed to solve the issue in this repo. It took me a bit more time than hoped for. As it turns out the request-promise-native library we are using is doing something with the body and will always try to translate the body to a string and is loading everything into memory. So after giving up on fixing that I did include the request library as that is a dependency of request-promise-native and low and behold things are making more sense now.

The whole stuff is also way better on memory consumption as it is no longer trying to download the full 2.5GB of my test file into memory. Would you have a change to test this repo against your scenario? I did manage to test with some various file sizes (from 10mb to 5GB) but before creating a PR and cleaning up a bit I would love some feedback! Thanks in advance.

@pnp/cli-for-microsoft-365-maintainers feel free to chime in as well 😉.

So minor update; managed to test some more 😉, but not quite there yet. We are using a custom implementation for all our requests in the request.ts file. I hacked in a .pipe() there and now I do get the file written to disk but somewhere the response is still cast, or transformed, as a string. I still get the same error (but now only after the file is saved to disk). Debugging is a bit hard since it does require at least a 2.1 GB file 😊. Will try to see if I can come up with a better approach on Monday.

Hi! Yes, the terminal commend is this (calling from python):

cmd = f"spo file get --webUrl '{self.__args.office365_url}' --url '{target_path}' --asFile --path '{download_path}'"

Darn, you are absolutely right! Will update tests and do PR (have to do some rework due to our latest changes but will get them in asap)

@appieschot if I look at the current documentation, it already states that path is required when using asFile

Hi @tobyatgithub i have a few days off in the run up to ignite, hope to work on it at the end of the week based on your replies. Sorry for the delay! Been busy remodeling our home.

Oh thats my bad! I never ran the tests, those are not updated yet. Could you skip the test for now so after npm run build just to the npm link. The tests are failing because I have changed the logic of saving with the --asFile parameter. I will go through the tests asap, but the code it self is correct so you can ignore the test for the first run 😃. Sorry for the inconvenience!

Hi Patrick, I think you are right! Sorry for misplacing this issue. It will be great if you can transfer it! Thank you and appreciate it 😮