langchainjs: Stream ConversationalRetrievalQAChain is not working

I’m using a pinecone vector with OpenAIChat model in a ConversationalRetrievalQAChain, it works as intended when im not streaming the output however if I change the OpenAIChat to streaming true as per the below code snippet it fails with the following error “TypeError: stream.getReader is not a function”

 const model = new OpenAIChat({
        streaming: true,
  callbackManager: CallbackManager.fromHandlers({
    async handleLLMNewToken(token: string) {
      console.log(token);
    },
  }),
    });

This is in a nextjs API route

About this issue

Original URL
State: closed
Created a year ago
Comments: 15 (1 by maintainers)

Commits related to this issue

Fix streaming errors in Node (Express, Next, etc.) environments (#2132) * Update event-source-parse.ts Fix streaming errors as seen in #548 * Fix TypeScript errors, make sure we send a promise ... — committed to langchain-ai/langchainjs by rishi-raj-jain a year ago

Most upvoted comments

I think i find why it didnt work, it should be the misuse the origin web env “fetch” and “node-fetch”. Check the detail here https://stackoverflow.com/questions/57664058/response-body-getreader-is-not-a-function.

Then I change the dep file event-source-parse getBytes function like this , problem solved.

export async function getBytes(stream, onChunk) {
    // const reader = stream.getReader();
    // let result;
    // // eslint-disable-next-line no-cond-assign
    // while (!(result = await reader.read()).done) {
    //     onChunk(result.value);
    // }
    stream.on('readable', () => {
        let chunk;
        while (null !== (chunk = stream.read())) {
            onChunk(chunk);
        }
    }
    );
}

mrslimslim on May 4, 2023

@dqbd do you want to have a look at this one? Would be pretty strange if nextjs/vercel didn’t support the ReadableStream web standard? Or is that people are running this in Node 16 without realising and Node 16 doesn’t support ReadableStream

nfcampos on May 11, 2023

This solution is fine! Can next version adapter browser fetch (node v18+) and node-fetch (node v16) ?

zhaoo on Jul 4, 2023

I think i find why it didnt work, it should be the misuse the origin web env “fetch” and “node-fetch”. Check the detail here https://stackoverflow.com/questions/57664058/response-body-getreader-is-not-a-function.

Then I change the dep file event-source-parse getBytes function like this , problem solved.
export async function getBytes(stream, onChunk) {
    // const reader = stream.getReader();
    // let result;
    // // eslint-disable-next-line no-cond-assign
    // while (!(result = await reader.read()).done) {
    //     onChunk(result.value);
    // }
    stream.on('readable', () => {
        let chunk;
        while (null !== (chunk = stream.read())) {
            onChunk(chunk);
        }
    }
    );
}

This solves the issue. I had the same problem while trying to stream the response from the /experimental/autogpt function with node v17.9.1 in my local environment. getReader() function is not supported by Node.js node-fetch package.

nsoticek on Jun 26, 2023