We were running into some of the same issues. We try to download files via XMLHttpRequest, and write the contents via a fileWriter. Writing files larger than a few megabytes would consistently crash the whole application. We switched to writing the files in chunks, which alleviated the problem, but I was still confused why writing files less than 50MiB would result in crashes or white-screens.

I profiled with chromes debugging tools, and found that the garbage collector is running like crazy, and the function uses a lot of memory. You can see here, that while invoking .writeFile that the memory usage goes from 11MiB to about 40MiB, even though we are writing 1MiB chunks of binary data.

As already mentioned by @breautek, this part in cordova is really slow: args[i] = base64.fromArrayBuffer(args[i]);

I wanted to understand what is happening here: Taking a look at the base64 module in cordova-common:

function uint8ToBase64 (rawData) {
    var numBytes = rawData.byteLength;
    var output = '';
    var segment;
    var table = b64_12bitTable();
    for (var i = 0; i < numBytes - 2; i += 3) {
        segment = (rawData[i] << 16) + (rawData[i + 1] << 8) + rawData[i + 2];
        output += table[segment >> 12];
        output += table[segment & 0xfff];
    }
    if (numBytes - i === 2) {
        segment = (rawData[i] << 16) + (rawData[i + 1] << 8);
        output += table[segment >> 12];
        output += b64_6bit[(segment & 0xfff) >> 6];
        output += '=';
    } else if (numBytes - i === 1) {
        segment = (rawData[i] << 16);
        output += table[segment >> 12];
        output += '==';
    }
    return output;
}

Ouch. JavaScript has immutable strings. Using += to concat the output creates new strings in memory all the time. This means if we need to convert n bytes to base64, we create at least 2 * n/3 new strings, each bigger than the last. This ends up costing O(n^2) bytes of memory. ( “AA”, “AABB”, “AABBCC” … all the way to the full string )

Note that if you want to write 1MiB, you would have to allocate several gigabytes of memory!

This explains the memory usage, and why writing binary files is slow. The garbage collector needs to clean up millions of strings, and if it can’t keep up, we run out of memory.

This not only affects writing files, but ANY cordova plugin that wants to pass binary data from JS to the native layer.

The conversion via FileReader.readAsDataURL that @digaus used is infinitely better than the cordova-common algorithm.

I thought it would be reasonable to support FileWriter.write(buffer: ArrayBuffer), and let it do the chunking and conversion to base64, to bypass the cordova algorithm. Then users of this plugin would not need to implement custom chunking code.

So I started to change the FileWriter.js code. The files are written to disk, but I must have made a mistake somewhere, because the files seem corrupted ( I have removed the prefix from the readAsDataURL result ).

I am also not sure why @digaus solution works. When I pass a string to FileWriter.write(), it should write the strings as-is to disk, instead of interpreting it as binary data encoded in base64? In www/FileWriter.js, we need to pass isBinary = true here:

}, 'File', 'write', [this.localURL, data, this.position, isBinary]);

In order to get LocalFileSystem.java writeToFileAtURL() to decode it to binary data.

Does FileSystem.writeFile behave differently?

I will spend some more time on this. I would really like to solve this issue and make a PR, once I get this to work.

@kputh This cant be an issue from cordova-common. cordova-common is not bundled with any app.

From the content of this thread, it sounds like you might mean cordova-js. This file specifically: https://github.com/apache/cordova-js/blob/master/src/common/base64.js

Thanks for the detailed analysis and potential fix(es) @LightMind - looking forward to hearing more from you.

I have managed to change FileWriter, such that it can convert ArrayBuffers to base64 encoded strings itself.

To get the correct base64 encoded string, it is important to call FileReader.readAsDataURL in the right way;

   fileReader.readAsDataURL(
        new Blob([arrayBuffer], {
            type: 'application/octet-binary'
        })
    );

Otherwise you would encode the string representation of the ArrayBuffer, which is not at all what you want. Link to the changed file

I have used this to write 1MiB and smaller files, and the performance looks better for now. Especially the memory usage for each file or chunk is only about twice the filesize/chunksize. There is less garbage collection happening.

I will test this with bigger files and measure the timing in comparison to the original version, to see if it actually performs better, and implement chunking, if that seems necessary.

File transer plugin was somewhat “undeprecated”, was upgraded to be available for latest cordova and shows good performance and results. Solved most of the issues my app was experiencing and it can also download when the app is not inn the front which is a nice plus. I recommend using it.

@breautek

We can convert the blob into chunks and write the file in parts:

   private async convertToBase64Chunks(blob: Blob, size: 3 | 6 | 9 | 12 | 15 | 18, writeChunk: (value: string, first?: boolean) => Promise<void>): Promise<void> {
       const chunkSize: number = 1024 * 1024 * size;
       const blobSize: number = blob.size;
       while (blob.size > chunkSize) {
           const value: string = await this.convertToBase64(blob.slice(0, chunkSize));
           await writeChunk(blobSize === blob.size ? value : value.split(',')[1], blobSize === blob.size);
           blob = blob.slice(chunkSize);
       }
       const lastValue: string = await this.convertToBase64(blob.slice(0, blob.size));
       await writeChunk(lastValue.split(',')[1], blobSize === blob.size);
       blob = blob.slice(blob.size);
   }


   private convertToBase64(blob: Blob): Promise<string> {
       return new Promise((resolve: (base64: string) => void, reject: () => void): void =>  {
           let reader: FileReader = new FileReader();
           const realFileReader: FileReader = reader['_realReader'];
           if (realFileReader) {
               reader = realFileReader;
           }
           reader.onerror = (err: any): void => {
               console.log(err);
               reject();
           };
           reader.onload = (): void => {
               resolve(reader.result as string);
           };
           reader.readAsDataURL(blob);
       });
   }

writeChunk is the asynchronus callback where we can call the write method of the plugin. first indicates the first part which also has the encoding prefix (but I think this is dropped anyways).

Implemented this in a Capacitor project which works very nicely:

            await this.convertToBase64Chunks(blob, chunkSize, async (value: string, first: boolean): Promise<void> => {
                if (first) {
                    await Filesystem.writeFile({
                        path: this.folder + '/' + fileName,
                        directory: FilesystemDirectory.Data,
                        data: value,
                    });
                } else {
                    await Filesystem.appendFile({
                        path: this.folder + '/' + fileName,
                        directory: FilesystemDirectory.Data,
                        data: value,
                    });
                }
            });

Edit: 20MB took arround 2 seconds to write

I think i cant make cordova calls from worker

You are correct, this is something that cordova doesn’t currently support.

so i need to pass my file to worker, convert it to base64 in worker, pass it back to main and call save()

You can try this, but I’m not certain if that will help as there is a bottleneck of copying the data to and from the worker.

If you’re downloading a file and saving it to disk, you’re best/quickest work around is probably using the File Transfer Plugin despite it being deprecated. Because it downloads the file and saves it to the filesystem all on the native side.

The primary reason why the transfer plugin is deprecated to my understanding is because it is possible to download binary files using XMLHttpRequest (however, then you can run into performance/efficiency issues like you are currently experiencing). Despite the plugin is being deprecated, I still use it to upload large files to my server, and that part is at least working for me still.

Thanks, I’ll try to take a quick look at this this weekend and report what I find on my devices.

Hello again! I made a sample reproduction app, it downloads 9mb png file and save it on disc, this cause around 4sec lag on all devices i tested.

Click on load button for load file, then click save button for save. =)