inspect_ai: Non-deterministic error in `proto` package when using Google models

This appears to happen non-deterministically, but often enough to be a big problem (have encountered it ~5 times).

Likely related to Google’s protocol buffers.

Traceback

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 62, in <module>
    click_cli()
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 58, in click_cli
    return cli(task_path, provider_path, model_name, airtable_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 45, in cli
    log = eval(task, provider_cls, model_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/benchmarks/benchmarks/bench/__init__.py", line [90](https://github.com/epoch-research/benchmarks/actions/runs/13284823215/job/37091147615#step:7:91), in eval
    logs = inspect_ai.eval(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 165, in eval
    return task_display().run_task_app(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_display/plain/display.py", line 45, in run_task_app
    return asyncio.run(main)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 416, in eval_async
    await eval_run(
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 214, in eval_run
    return await run_single(task_run_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 237, in run_single
    return await asyncio.gather(*asyncio_tasks)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 324, in task_run
    sample_results = await asyncio.gather(*sample_coroutines)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 733, in task_run_sample
    error = handle_error(ex)
            ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 542, in handle_error
    err = sample_error(ex)
          ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/error.py", line 22, in __call__
    raise ex
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 653, in task_run_sample
    error = handle_error(ex)
            ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 542, in handle_error
    err = sample_error(ex)
          ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/error.py", line 22, in __call__
    raise ex
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 597, in task_run_sample
    state = await plan(state, generate)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_plan.py", line 105, in __call__
    state = await solver(state, generate)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_solver.py", line 278, in solve
    return await generate(state, tool_calls=tool_calls, cache=cache, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 243, in generate
    return await task_generate(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/generate.py", line 31, in task_generate
    state.output = await model.generate(
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 261, in generate
    return await self._generate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 441, in _generate
    model_output = await generate()
                   ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
    return await copy(fn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
    do = await self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
    result = await action(retry_state)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
    return call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
    result = await fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 388, in generate
    result = await self.api.generate(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 190, in generate
    choices=completion_choices_from_candidates(response.candidates),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 560, in completion_choices_from_candidates
    return [completion_choice_from_candidate(candidate) for candidate in candidates]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 560, in <listcomp>
    return [completion_choice_from_candidate(candidate) for candidate in candidates]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen _collections_abc>", line 9[93](https://github.com/epoch-research/benchmarks/actions/runs/13284823215/job/37091147615#step:7:94), in __iter__
  File "/usr/local/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 125, in __getitem__
    return self._marshal.to_python(self._pb_type, self.pb[key])
                                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/proto/utils.py", line 50, in inner
    self._cached_values[fx.__name__] = fx(self)
                                       ^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 116, in _pb_type
    canary = copy.deepcopy(self.pb).add()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'add'

Environment

Full lockfile

Uploaded as .txt because GitHub won’t allow arbitrary file types

poetry.lock.txt

Selected dependencies

❯ poetry show protobuf
 name         : protobuf 
 version      : 4.25.5   
 description  :          

required by
 - google-ai-generativelanguage >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - google-api-core >=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0.dev0
 - google-cloud-aiplatform >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - google-cloud-resource-manager >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - google-generativeai *
 - googleapis-common-protos >=3.20.2,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0.dev0
 - grpc-google-iam-v1 >=3.20.2,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
 - grpcio-status >=4.21.6
 - mlflow-skinny >=3.12.0,<6
 - proto-plus >=3.19.0,<6.0.0dev
 - sagemaker >=3.12,<5.0
❯ poetry show google-generativeai
 name         : google-generativeai                                           
 version      : 0.8.3                                                         
 description  : Google Generative AI High level API client library and tools. 

dependencies
 - google-ai-generativelanguage 0.6.10
 - google-api-core *
 - google-api-python-client *
 - google-auth >=2.15.0
 - protobuf *
 - pydantic *
 - tqdm *
 - typing-extensions *

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (9 by maintainers)

Most upvoted comments

Got it. Yes, in that case my view would be to fail loudly.

Yes you can say stop_reason="content_filter". The old API had no info about why there are no choices (but did it anyway sometimes). I thought it better to classify it as “unknown” since it’s kind of speculation that its a content filter. With the new API they have added a field indicating that it is a refusal along with the reason. Even with the new API though its debatable whether to fail loudly or not, as if there are other reasons they might not return completions – if we raise we make it impossible to complete an eval on that dataset without an error. As they’ve already demonstrated that they are willing to have “no completions” occur w/ no explanation, there is no guarantee they won’t do this in the future. It would be a better world if all of the model APIs were exhaustively specified and fully deterministic but they simply aren’t, so we need to make some provision to navigate around the vagaries, inconsistencies, and bugs in these services.

Yeah, I saw that it was fine. The MutableSequence type doesn’t have a .sort() method (which we were calling without an exception) so my best guess is that it was a list anyway. In any case, I did update the types and move to the sorted() method. The speculative fix is really to check for a falsey value on candidates (which we’d get for a null value or empty list).

We’ve noticed that in some cases Google has started returning 0 completion choices when there is a refusal. It may be that their type declarations are out of sync with their service (or perhaps they are a/b testing a new service endpoint and that’s where the declarations are out of sync).

Further, Google maddeningly doesn’t produce type declarations for their genai packages — I noticed that under the hood the candidates type is MutableSequence[Candidate] rather than list[Candidate] (which is what our helper function was taking). Unlikely this is of consequence but worth running down every angle.

Finally, Google is in the course of introducing a brand new Python client for Gemini. They have said it is “not ready for production”. We nevertheless have an implementation of it internally that we are hoping to promote soon (but we’ve observed numerous issues with it).

Here is a speculative fix for the candidates issues: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1302