dspy: ValueError: Expected dict_keys(['answer']) but got dict_keys([])

I run the official code example in intro.ipynb:

import dspy

lm = dspy.LM(model='openai/default', api_key=" ", api_base=" ",temperature=0.9, max_tokens=3000,)
colbertv2_wiki17_abstracts = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')

dspy.settings.configure(lm=lm, rm=colbertv2_wiki17_abstracts)

from dspy.datasets import HotPotQA

# Load the dataset.
dataset = HotPotQA(train_seed=1, train_size=20, eval_seed=2023, dev_size=50, test_size=0)

# Tell DSPy that the 'question' field is the input. Any other fields are labels and/or metadata.
trainset = [x.with_inputs('question') for x in dataset.train]
devset = [x.with_inputs('question') for x in dataset.dev]

len(trainset), len(devset)

train_example = trainset[0]
print(f"Question: {train_example.question}")
print(f"Answer: {train_example.answer}")

dev_example = devset[18]
print(f"Question: {dev_example.question}")
print(f"Answer: {dev_example.answer}")
print(f"Relevant Wikipedia Titles: {dev_example.gold_titles}")


class BasicQA(dspy.Signature):
    """Answer questions with short factoid answers."""

    question = dspy.InputField()
    answer = dspy.OutputField(desc="often between 1 and 5 words")
    
# Define the predictor.
generate_answer = dspy.Predict(BasicQA)

# Call the predictor on a particular input.
pred = generate_answer(question=dev_example.question)

# Print the input and the prediction.
print(f"Question: {dev_example.question}")
print(f"Predicted Answer: {pred.answer}")

ERROR I get

Traceback (most recent call last):
  File "/mnt/rangehow/dspy/1.py", line 39, in <module>
    pred = generate_answer(question=dev_example.question)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/rangehow/dspy/dspy/predict/predict.py", line 98, in __call__
    return self.forward(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/rangehow/dspy/dspy/predict/predict.py", line 131, in forward
    completions = v2_5_generate(lm, config, signature, demos, kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/rangehow/dspy/dspy/predict/predict.py", line 231, in v2_5_generate
    return adapter(lm, lm_kwargs=lm_kwargs, signature=signature, demos=demos, inputs=inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/rangehow/dspy/dspy/adapters/base.py", line 11, in __call__
    value = self.parse(signature, output)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/rangehow/dspy/dspy/adapters/chat_adapter.py", line 58, in parse
    raise ValueError(f"Expected {signature.output_fields.keys()} but got {fields.keys()}")
ValueError: Expected dict_keys(['answer']) but got dict_keys([])

The outputs from LLM, I print it out:

['answer: American\n[[ ## completed ## ]]']

About this issue

Most upvoted comments

The whole concept of DSPy as incremental improvement through optimization is very powerful. In that vein, couldn’t an error in formatting should be treated as the equivalent of a wrong answer in evaluate or optimization? Then through iterations these could be reduced by getting the model to output better structured responses?

I’d be happy to contribute but not sure how much of the DSPy python modules would need to be changed…

For us folks on a budget or running everything locally it seems like a super interesting way to try and improve outputs. Constrained grammars as suggested by @dvs23 seems like it could be one way to tackle this. Perhaps that is already in the works?

Any news on this? I still have this problem with DSPy 2.6.2 and primarily phi4 (although every model can in principle fail to generate the correct output format, of course).

I think this is something that should be covered by the max_errors parameter, as currently a single failure of a model to provide correctly structured outputs at that point crashed the whole MIPROv2 optimization for me:

Traceback (most recent call last):
  File "/some/path/venv/lib/python3.10/site-packages/dspy/adapters/base.py", line 30, in __call__
    value = self.parse(signature, output)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/utils/callback.py", line 234, in wrapper
    return fn(instance, *args, **kwargs)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/adapters/chat_adapter.py", line 86, in parse
    raise ValueError(f"Expected {signature.output_fields.keys()} but got {fields.keys()}")
ValueError: Expected dict_keys(['proposed_instruction']) but got dict_keys([])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/some/path/src/llm/compositionality/fewshot.py", line 296, in fewshot_mipro
    optimized_program = teleprompter.compile(
  File "/some/path/venv/lib/python3.10/site-packages/dspy/teleprompt/mipro_optimizer_v2.py", line 173, in compile
    instruction_candidates = self._propose_instructions(
  File "/some/path/venv/lib/python3.10/site-packages/dspy/teleprompt/mipro_optimizer_v2.py", line 451, in _propose_instructions
    instruction_candidates = proposer.propose_instructions_for_program(
  File "/some/path/venv/lib/python3.10/site-packages/dspy/propose/grounded_proposer.py", line 358, in propose_instructions_for_program
    self.propose_instruction_for_predictor(
  File "/some/path/venv/lib/python3.10/site-packages/dspy/propose/grounded_proposer.py", line 410, in propose_instruction_for_predictor
    proposed_instruction = instruction_generator.forward(
  File "/some/path/venv/lib/python3.10/site-packages/dspy/propose/grounded_proposer.py", line 246, in forward
    instruct = self.generate_module_instruction(
  File "/some/path/venv/lib/python3.10/site-packages/dspy/utils/callback.py", line 234, in wrapper
    return fn(instance, *args, **kwargs)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/predict/predict.py", line 67, in __call__
    return self.forward(**kwargs)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/predict/predict.py", line 97, in forward
    completions = adapter(lm, lm_kwargs=config, signature=signature, demos=demos, inputs=kwargs)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/adapters/base.py", line 45, in __call__
    return JSONAdapter()(lm, lm_kwargs, signature, demos, inputs)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/adapters/json_adapter.py", line 61, in __call__
    value = self.parse(signature, output)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/utils/callback.py", line 234, in wrapper
    return fn(instance, *args, **kwargs)
  File "/some/path/venv/lib/python3.10/site-packages/dspy/adapters/json_adapter.py", line 95, in parse
    fields = {k: v for k, v in fields.items() if k in signature.output_fields}
AttributeError: 'list' object has no attribute 'items'

At the same time, I feel like this is something that should be handled using (grammar-)constrained decoding (see [1,2], repos: [3,4]), as the expected output structure and especially JSON is relatively well-defined. I could maybe help with that, we ([2, 4]) already did some things with GCD in the past.

I know this is not feasible in every environment, as you need logit-level access to the LM output, but in some it should be possible. For example, llama.cpp even supports grammar-constrained decoding already [5] and Ollama is apparently working on support for returning logits/logprobs via the API as well [6]. Maybe one could also leverage the already-existing DSPy Assertions for this?

[1] https://doi.org/10.18653/v1/2023.emnlp-main.674 [2] https://doi.org/10.3389/frai.2024.1406857 [3] https://github.com/epfl-dlab/transformers-CFG [4] https://github.com/ag-sc/Clinical-Trial-IE-GCD [5] https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md [6] https://github.com/ollama/ollama/issues/2415

Sweet thanks for sending this. Will look more into it tomorrow

I think a nice addition that could work is auto-fixing the output by the model itself. I.e. if the field is wrongly structured ask the model to fix it. This should be straight forward to implement.