dspy: ValueError: Expected dict_keys(['answer']) but got dict_keys([])
I run the official code example in intro.ipynb:
import dspy
lm = dspy.LM(model='openai/default', api_key=" ", api_base=" ",temperature=0.9, max_tokens=3000,)
colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')
dspy.settings.configure(lm=lm, rm=colbertv2_wiki17_abstracts)
from dspy.datasets import HotPotQA
# Load the dataset.
dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)
# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.
trainset = [x.with_inputs('question') for x in dataset.train]
devset = [x.with_inputs('question') for x in dataset.dev]
len(trainset), len(devset)
train_example = trainset[0]
print(f"Question: {train_example.question}")
print(f"Answer: {train_example.answer}")
dev_example = devset[18]
print(f"Question: {dev_example.question}")
print(f"Answer: {dev_example.answer}")
print(f"Relevant Wikipedia Titles: {dev_example.gold_titles}")
class BasicQA(dspy.Signature):
"""Answer questions with short factoid answers."""
question = dspy.InputField()
answer = dspy.OutputField(desc="often between 1 and 5 words")
# Define the predictor.
generate_answer = dspy.Predict(BasicQA)
# Call the predictor on a particular input.
pred = generate_answer(question=dev_example.question)
# Print the input and the prediction.
print(f"Question: {dev_example.question}")
print(f"Predicted Answer: {pred.answer}")
ERROR I get
Traceback (most recent call last):
File "/mnt/rangehow/dspy/1.py", line 39, in <module>
pred = generate_answer(question=dev_example.question)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/rangehow/dspy/dspy/predict/predict.py", line 98, in __call__
return self.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/rangehow/dspy/dspy/predict/predict.py", line 131, in forward
completions = v2_5_generate(lm, config, signature, demos, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/rangehow/dspy/dspy/predict/predict.py", line 231, in v2_5_generate
return adapter(lm, lm_kwargs=lm_kwargs, signature=signature, demos=demos, inputs=inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/rangehow/dspy/dspy/adapters/base.py", line 11, in __call__
value = self.parse(signature, output)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/rangehow/dspy/dspy/adapters/chat_adapter.py", line 58, in parse
raise ValueError(f"Expected {signature.output_fields.keys()} but got {fields.keys()}")
ValueError: Expected dict_keys(['answer']) but got dict_keys([])
The outputs from LLM, I print it out:
['answer: American\n[[ ## completed ## ]]']
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 35
The whole concept of DSPy as incremental improvement through optimization is very powerful. In that vein, couldn’t an error in formatting should be treated as the equivalent of a wrong answer in evaluate or optimization? Then through iterations these could be reduced by getting the model to output better structured responses?
I’d be happy to contribute but not sure how much of the DSPy python modules would need to be changed…
For us folks on a budget or running everything locally it seems like a super interesting way to try and improve outputs. Constrained grammars as suggested by @dvs23 seems like it could be one way to tackle this. Perhaps that is already in the works?
Any news on this? I still have this problem with DSPy 2.6.2 and primarily phi4 (although every model can in principle fail to generate the correct output format, of course).
I think this is something that should be covered by the max_errors parameter, as currently a single failure of a model to provide correctly structured outputs at that point crashed the whole MIPROv2 optimization for me:
At the same time, I feel like this is something that should be handled using (grammar-)constrained decoding (see [1,2], repos: [3,4]), as the expected output structure and especially JSON is relatively well-defined. I could maybe help with that, we ([2, 4]) already did some things with GCD in the past.
I know this is not feasible in every environment, as you need logit-level access to the LM output, but in some it should be possible. For example, llama.cpp even supports grammar-constrained decoding already [5] and Ollama is apparently working on support for returning logits/logprobs via the API as well [6]. Maybe one could also leverage the already-existing DSPy Assertions for this?
[1] https://doi.org/10.18653/v1/2023.emnlp-main.674 [2] https://doi.org/10.3389/frai.2024.1406857 [3] https://github.com/epfl-dlab/transformers-CFG [4] https://github.com/ag-sc/Clinical-Trial-IE-GCD [5] https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md [6] https://github.com/ollama/ollama/issues/2415
Sweet thanks for sending this. Will look more into it tomorrow
I think a nice addition that could work is auto-fixing the output by the model itself. I.e. if the field is wrongly structured ask the model to fix it. This should be straight forward to implement.