dspy: 500 internal error when calling Azure OpenAI GPT4 with dspy
Hello all!
Currently experiencing 500 internal errors when using DSPY with Azure OpenAI gpt-4.
I have been using the Azure OpenAI client to ping a gpt-4 deployment, and am consistently getting: Internal service error 500. I’ve also pinged using litellm for a manual prompting pipeline I’ve set up alongside it and have not seen any issues / errors returned. indicating it’s not an Azure side issue - nor a ratelimit issue since I did them right back to back but not together.
I’ve checked that both dspy’s azure openai client and my litellm pipeline correctly sends a message with keys ‘role’ and ‘content’… I can’t seem to find the issue.
The signature and module setup I have is as follows:
class QuestionAnswerSignature(dspy.Signature):
context: str = dspy.InputField()
question: str = dspy.InputField()
choices: list[str] = dspy.InputField(format=list)
answer: QA_output = dspy.OutputField(desc="key-value pairs")
class QuestionAnswer(dspy.Module):
def __init__(self):
super().__init__()
self.pick_answer = dspy.Predict(QuestionAnswerSignature)
def forward(self, context, question, choices):
output = self.pick_answer(
context = context,
question=question,
choices=choices
)
# suggest that length should be less than 5 tokens
dspy.Assert(len(output) < 100)
return output
And the actual instantiation of the client I have is as shown below.
gpt_dspy = AzureOpenAI(
model=model_name,
api_base = gpt_client.api_base,
api_version = gpt_client.api_version,
api_key = gpt_client._get_token(),
model_type = 'chat',
)
dspy.settings.configure(lm=gpt_dspy)
The request is sent correctly to the right address, as can be seen by the 500 + the logs also indicate the right URL is being used.
The only other thing I can think of is that either certain keyword arguments are missing (which I would assume would give a 400…) or that the prompt text generated by dspy somehow throws an error on the gpt side.
Is there currently known behavior of Azure openai GPT4 not working with DSPY?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 21
You need to also provide deployment_id=“your deployment name” in the instantiation of the client