distributed: Series.values.compute() leads to "TypeError: can't concat buffer to bytearray"

Running a local dask-scheduler + dask-worker pair, the following code leads to a crash of the worker:

import dask.dataframe as dd
from dask.distributed import Executor
e = Executor('127.0.0.1:8786', set_as_default=True)
df = pd.DataFrame({"A": [1, 2, 3] * 10})
ddf = dd.from_pandas(df, npartitions=3)
ddf["A"].values.compute()

The worker crashes with:

Traceback (most recent call last):
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/core.py", line 259, in handle_comm
    result = yield result
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/worker.py", line 439, in get_data
    compressed = yield comm.write(msg)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/gen.py", line 292, in wrapper
    result = func(*args, **kwargs)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/distributed/comm/tcp.py", line 196, in write
    stream.write(frame)
  File "/home/PHI-TPS/fkeller/.virtualenvs/dask/local/lib/python2.7/site-packages/tornado/iostream.py", line 395, in write
    self._write_buffer += data
TypeError: can't concat buffer to bytearray

Other computations like ddf["A"].compute() or even ddf.values.compute() work fine though.

About this issue

Original URL
State: closed
Created 7 years ago
Comments: 23 (22 by maintainers)

Most upvoted comments

For the record:

$ ./python -V
Python 2.7.3
$ ./python -c "b = bytearray(); b+= buffer(b'123'); print(b)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: can't concat buffer to bytearray

$ ./python -V
Python 2.7.4
$ ./python -c "b = bytearray(); b+= buffer(b'123'); print(b)"
123

pitrou on Jun 20, 2017

Lets wait for @pitrou to take a look at the error. He is more familiar with the networking stack than I am (I think he wrote the bytearray code in tornado) and may have thoughts. Unfortunately I think he’s on vacation until Monday.

mrocklin on Jun 16, 2017