prefect: Handling non-pickleable exceptions

Not sure if you would agree that this is a bug rather than unsupported behavior.

Google libraries tend to include non-pickleable Client objects in the exceptions they raise (who knows why). So the following flow run locally gives a nice clear error:

from google.cloud import bigquery

@task
def bq_fail():
    bq = bigquery.Client()
    bq.query('select 1 from nonexistent_table').result()
    return True

with Flow('f') as f:
    bq_fail()

f.run()

...
google.api_core.exceptions.BadRequest: 400 Table name "nonexistent_table" missing dataset while no default dataset is set in the request.

(job ID: 69b11ec8-312b-49c4-91c9-aae701b7b8cf)

  -----Query Job SQL Follows-----

    |    .    |    .    |    .    |
   1:select 1 from nonexistent_table
    |    .    |    .    |    .    |
[2020-03-20 18:21:39,312] INFO - prefect.TaskRunner | Task 'bq_fail': finished task run for task with final state: 'Failed'
INFO:prefect.TaskRunner:Task 'bq_fail': finished task run for task with final state: 'Failed'
[2020-03-20 18:21:39,313] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
INFO:prefect.FlowRunner:Flow run FAILED: some reference tasks failed.

but running on a Dask client gives a much crazier failure:

import distributed
from prefect.engine.executors import DaskExecutor
client = distributed.Client()
f.run(executor=DaskExecutor(client.scheduler.address))

...
Could not serialize object of type Failed.
Traceback (most recent call last):
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 38, in dumps
    result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
AttributeError: Can't pickle local object 'if_exception_type.<locals>.if_exception_type_predicate'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 191, in serialize
    header, frames = dumps(x, context=context) if wants_context else dumps(x)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/serialize.py", line 58, in pickle_dumps
    return {"serializer": "pickle"}, [pickle.dumps(x)]
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/distributed/protocol/pickle.py", line 51, in dumps
    return cloudpickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 1125, in dumps
    cp.dump(obj)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 482, in dump
    return Pickler.dump(self, obj)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
  File "/Users/brett/model/.venv/lib/python3.7/site-packages/google/cloud/client.py", line 144, in __getstate__
    "Clients have non-trivial state that is local and unpickleable.",
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.

This is actually a very abbreviated version of the actual error: the real output contains like 10 variations of the above and sort of swamps all other output in the logs.

I understand that Google is the underlying culprit here, but it seems like Prefect could handle this case more gracefully, maybe by just catching a PickleError and re-raising as something more informative…?

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 16 (3 by maintainers)

Commits related to this issue

Merge pull request #2178 from PrefectHQ/docs-notifications Docs: Notifications — committed to PrefectHQ/prefect by zanieb 2 years ago

Most upvoted comments

No progress yet - someone will comment here when things have changed.

jcrist on Jul 1, 2020

It’s not just the latest exception that is being stored, but rather some kind of exception chain that includes the previous boto exception. (I have no idea whether this is even a possibility)

Ah, good catch. Python supports chained exceptions, so the previous exception would leave stuff on the __cause__ and __context__ attributes of the raised MyBotoException. Can you try this one?

from functools import wraps

def fix_boto_exceptions(func):
    @wraps(func)
    def inner(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except BaseBotoException as exc:    # I don't know what this class is called, but you probably do
            raise MyBotoException(str(exc)) from None
    return inner

@task
@fix_boto_exceptions
def mytask(...):
    some_boto_thing()
    some_other_boto_thing()

Still not ideal, but I believe this should work for you.

jcrist on Jun 24, 2020