langchainjs: ConversationalRetrievalQAChain factory method only supports QA stuff chain, fails on long inputs

Here is a snippet of my code:

const loader = new PDFLoader('./基于贝叶斯优化的无标签网络剪枝算法.pdf')
const pdf = await loader.load()
const textSplitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 0
})
const docs = await textSplitter.splitDocuments(pdf)
const vectorStore = await PineconeStore.fromDocuments(
  docs,
  new OpenAIEmbeddings({
    openAIApiKey: process.env.OPEN_API_KEY
  }),
  {
    pineconeIndex
  }
)`
`const vectorStore = await PineconeStore.fromExistingIndex(
  new OpenAIEmbeddings({
    openAIApiKey: process.env.OPEN_API_KEY
  }),
  { pineconeIndex }
)
/* Create the chain */
const chain = ConversationalRetrievalQAChain.fromLLM(
  model,
  vectorStore.asRetriever(),
  { verbose: true }
)
const question =
  '对于没有经过任何优化的高斯过程，n 个样本点时间复杂度大概是多少?'
try {
  // your async function here
  const res = await chain.call({ question, chat_history: [] })
  console.log(res.text)
} catch (error) {
  if (error.response) {
    console.log(error.response.status)
    console.log(error.response.data)
  } else {
    console.log(error.data.message)
  }
}

About this issue

Original URL
State: closed
Created a year ago
Comments: 16

Most upvoted comments

It’s okay, I’ve got it covered. I really appreciate your help!

Eventidus on Jun 3, 2023

Awesome! I’m going to leave this open though because this should be configurable.

jacoblee93 on May 29, 2023

The error message you’re seeing means that your prompt is too many tokens, and it looks like that’s happening because the ConversationalRetrievalQAChain.fromLLM method is only letting you use a specific type of chain with no option to change it:

https://github.com/hwchase17/langchainjs/blob/main/langchain/src/chains/conversational_retrieval_chain.ts#L156

For now, can you try creating the conversation chain similarly to this?

import { loadQAMapReduceChain } from "langchain/chains/load";
const question_generator_template = `Given the following conversation and a follow up question, rephrase the follow up question to be a standalone question.

Chat History:
{chat_history}
Follow Up Input: {question}
Standalone question:`;

const qa_template = `Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

Question: {question}
Helpful Answer:`;

const chain = new ConversationalRetrievalQAChain({
  retriever: vectorStore.asRetriever(),
  combineDocumentsChain: loadQAMapReduceChain(model),
  questionGeneratorChain: new LLMChain({
    prompt: question_generator_prompt,
    llm: model,
    verbose: true,
  }),
  verbose: true,
});

Basically, using the constructor manually with a map reduce chain instead of the default stuff chain: https://github.com/hwchase17/langchainjs/blob/main/langchain/src/chains/conversational_retrieval_chain.ts#L62

jacoblee93 on May 26, 2023