langchain: on_llm_start callback doesn't change the prompts sent to LLM

System Info

Python 3.10.12 langchain 0.0.233

Who can help?

@agola11

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

from typing import Dict, Union, Any, List

from langchain.callbacks.base import BaseCallbackHandler
from langchain.schema import AgentAction
from langchain.agents import AgentType, initialize_agent, load_tools
from langchain.callbacks import tracing_enabled
from langchain.llms import OpenAI
from langchain.schema.agent import AgentAction
from langchain.schema.output import LLMResult
from uuid import UUID

class MyCustomHandlerOne(BaseCallbackHandler):
    def on_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any
    ) -> Any:
        print(f"on_llm_start {prompts}")
        prompts[0] = "Tell me a joke?"
        print(f"on_llm_start {prompts}")

import langchain
langchain.verbose = True
langchain.debug = True

handler1 = MyCustomHandlerOne()
llm = OpenAI(temperature=0, streaming=True, callbacks=[handler1], verbose=True)
print(llm.__call__("""Context : Mr. Carl Smith is a 31-year-old man who has been experiencing homelessness 
                   on and off for all his adult life.
                   Questioin: How old is Mr. Carl Smith?"""))

The output is

on_llm_start ['Context : Mr. Carl Smith is a 31-year-old man who has been experiencing homelessness \n                   on and off for all his adult life.\n                   Questioin: How old is Mr. Carl Smith?']
on_llm_start ['Tell me a joke?']
[llm/start] [1:llm:OpenAI] Entering LLM run with input:
{
  "prompts": [
    "Tell me a joke?"
  ]
}
[llm/end] [1:llm:OpenAI] [699.067ms] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "\n\nAnswer: Mr. Carl Smith is 31 years old.",
        "generation_info": {
          "finish_reason": "stop",
          "logprobs": null
        }
      }
    ]
  ],
  "llm_output": {
    "token_usage": {},
    "model_name": "text-davinci-003"
  },
  "run": null
}


Answer: Mr. Carl Smith is 31 years old.

Expected behavior

LLM will respond to the new prompt set by the callback