langchain: Fix hallucination problem with `SQLDatabaseChain`
Description
Based on my experiments with SQLDatabaseChain class, I have noticed that the LLM tends to hallucinate Answer if SQLResult field results as []. As I primarily used OpenAI as the LLM, I am not sure if this problem exists for other LLMs.
This is surely an undesirable outcome for users if the agent responds back with incorrect answers. With some prompt engineering, I think the following can be added to the default prompt to tackle this problem:
If the SQLResult is empty, the Answer should be "No results found". DO NOT hallucinate an answer if there is no result.
If there are better prompts to tackle this, that’s great too!
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 17 (3 by maintainers)
Great answer @ajndkr , solved my problem. Added your text before the user query, and I got as answer: SQLQuery: N/A(sqlite3.OperationalError) near “N”: syntax error [SQL: N/A]
captured this answer as an exception, and returned to the user : “I don´t know” .
Many thanks
@rubensmau for your use case, you can look into https://github.com/hwchase17/langchain/blob/ced412e1c12586a032eef937b97f195b9542c6d3/langchain/agents/agent_toolkits/sql/prompt.py#L14
Yeah since you said ‘the agent prefix’ I thought you referred to the agent. I think the agent does not have this problem because it has some kind of instruction not to invent if the tables do not give it an answer. I would go through with this for the chain 👍