solana-web3.js: Web3.js cannot handle chunked transfer encodings

Overview

Web3js throws an error when it processes a response from an RPC server with chunked transfer encoding: Invalid response body while trying to fetch [...]: Premature close. This happens because web3js is bundling a bad version of node-fetch. The bad version of node-fetch is 2.6.12 and later versions are also impacted. Version 2.6.7 does not have this problem. Here is the bug in node-fetch: https://github.com/node-fetch/node-fetch/issues/1219

It’s critical to support chunked transfer encodings since its part of the HTTP/1.1 spec. This bug is very limiting for RPC providers such as ourselves (Helius) trying to make performance optimizations.

Web3js users impacted by this bug can workaround by installing version 2.6.7 and overriding the fetch setting of the Connection object:

    const fetch = require('node-fetch');
    const connection = new Connection(connectionString, {
        commitment: 'confirmed',
        fetch: (url, init) => fetch(url, { ...init, compress: true }),
    });

Steps to reproduce

1. Create an RPC proxy that returns data with Transfer-Encoding: chunked. Alternatively you can use Helius RPCs, but we will soon be changing it due to this issue. You’re welcome to create a key for free at https://dev.helius.xyz.
2. Set web3js to 1.78.0 and run npm install.
3. Create a standard connection object (new Connection('https://rpc.helius.xyz?api-key-here'))
4. Make any RPC request.

Description of bug

All details provided in the overview.