inspect_ai: Non-deterministic error in `proto` package when using Google models
This appears to happen non-deterministically, but often enough to be a big problem (have encountered it ~5 times).
Likely related to Google’s protocol buffers.
Traceback
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 62, in <module>
click_cli()
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 58, in click_cli
return cli(task_path, provider_path, model_name, airtable_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/benchmarks/benchmarks/bench/cli/run_eval.py", line 45, in cli
log = eval(task, provider_cls, model_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/__w/benchmarks/benchmarks/bench/__init__.py", line [90](https://github.com/epoch-research/benchmarks/actions/runs/13284823215/job/37091147615#step:7:91), in eval
logs = inspect_ai.eval(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 165, in eval
return task_display().run_task_app(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_display/plain/display.py", line 45, in run_task_app
return asyncio.run(main)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 416, in eval_async
await eval_run(
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 214, in eval_run
return await run_single(task_run_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/run.py", line 237, in run_single
return await asyncio.gather(*asyncio_tasks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 324, in task_run
sample_results = await asyncio.gather(*sample_coroutines)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 733, in task_run_sample
error = handle_error(ex)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 542, in handle_error
err = sample_error(ex)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/error.py", line 22, in __call__
raise ex
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 653, in task_run_sample
error = handle_error(ex)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 542, in handle_error
err = sample_error(ex)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/error.py", line 22, in __call__
raise ex
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 597, in task_run_sample
state = await plan(state, generate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_plan.py", line 105, in __call__
state = await solver(state, generate)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/solver/_solver.py", line 278, in solve
return await generate(state, tool_calls=tool_calls, cache=cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/run.py", line 243, in generate
return await task_generate(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/_eval/task/generate.py", line 31, in task_generate
state.output = await model.generate(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 261, in generate
return await self._generate(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 441, in _generate
model_output = await generate()
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__
do = await self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 153, in iter
result = await action(retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tenacity/_utils.py", line 99, in inner
return call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__
result = await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_model.py", line 388, in generate
result = await self.api.generate(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 190, in generate
choices=completion_choices_from_candidates(response.candidates),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 560, in completion_choices_from_candidates
return [completion_choice_from_candidate(candidate) for candidate in candidates]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/inspect_ai/model/_providers/google.py", line 560, in <listcomp>
return [completion_choice_from_candidate(candidate) for candidate in candidates]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen _collections_abc>", line 9[93](https://github.com/epoch-research/benchmarks/actions/runs/13284823215/job/37091147615#step:7:94), in __iter__
File "/usr/local/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 125, in __getitem__
return self._marshal.to_python(self._pb_type, self.pb[key])
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/proto/utils.py", line 50, in inner
self._cached_values[fx.__name__] = fx(self)
^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 116, in _pb_type
canary = copy.deepcopy(self.pb).add()
^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'add'
Environment
Full lockfile
Uploaded as .txt because GitHub won’t allow arbitrary file types
Selected dependencies
❯ poetry show protobuf
name : protobuf
version : 4.25.5
description :
required by
- google-ai-generativelanguage >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
- google-api-core >=3.19.5,<3.20.0 || >3.20.0,<3.20.1 || >3.20.1,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0.dev0
- google-cloud-aiplatform >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
- google-cloud-resource-manager >=3.20.2,<4.21.0 || >4.21.0,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
- google-generativeai *
- googleapis-common-protos >=3.20.2,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0.dev0
- grpc-google-iam-v1 >=3.20.2,<4.21.1 || >4.21.1,<4.21.2 || >4.21.2,<4.21.3 || >4.21.3,<4.21.4 || >4.21.4,<4.21.5 || >4.21.5,<6.0.0dev
- grpcio-status >=4.21.6
- mlflow-skinny >=3.12.0,<6
- proto-plus >=3.19.0,<6.0.0dev
- sagemaker >=3.12,<5.0
❯ poetry show google-generativeai
name : google-generativeai
version : 0.8.3
description : Google Generative AI High level API client library and tools.
dependencies
- google-ai-generativelanguage 0.6.10
- google-api-core *
- google-api-python-client *
- google-auth >=2.15.0
- protobuf *
- pydantic *
- tqdm *
- typing-extensions *
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 16 (9 by maintainers)
Got it. Yes, in that case my view would be to fail loudly.
Yes you can say
stop_reason="content_filter". The old API had no info about why there are no choices (but did it anyway sometimes). I thought it better to classify it as “unknown” since it’s kind of speculation that its a content filter. With the new API they have added a field indicating that it is a refusal along with the reason. Even with the new API though its debatable whether to fail loudly or not, as if there are other reasons they might not return completions – if we raise we make it impossible to complete an eval on that dataset without an error. As they’ve already demonstrated that they are willing to have “no completions” occur w/ no explanation, there is no guarantee they won’t do this in the future. It would be a better world if all of the model APIs were exhaustively specified and fully deterministic but they simply aren’t, so we need to make some provision to navigate around the vagaries, inconsistencies, and bugs in these services.Yeah, I saw that it was fine. The
MutableSequencetype doesn’t have a.sort()method (which we were calling without an exception) so my best guess is that it was a list anyway. In any case, I did update the types and move to thesorted()method. The speculative fix is really to check for a falsey value oncandidates(which we’d get for a null value or empty list).We’ve noticed that in some cases Google has started returning 0 completion choices when there is a refusal. It may be that their type declarations are out of sync with their service (or perhaps they are a/b testing a new service endpoint and that’s where the declarations are out of sync).
Further, Google maddeningly doesn’t produce type declarations for their genai packages — I noticed that under the hood the
candidatestype isMutableSequence[Candidate]rather thanlist[Candidate](which is what our helper function was taking). Unlikely this is of consequence but worth running down every angle.Finally, Google is in the course of introducing a brand new Python client for Gemini. They have said it is “not ready for production”. We nevertheless have an implementation of it internally that we are hoping to promote soon (but we’ve observed numerous issues with it).
Here is a speculative fix for the candidates issues: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1302