modin: BUG: passing in large data object to DataFrame apply() function resulting in SegFault

@anmyachev I can make it to work by not setting an RAY_ADDRESS, Modin/Ray seems to be able to pick up the address when I am running the client code on one of the Ray nodes. However, it is worth noting that I am seeing different ports after this change, before it was on port 10001, and now 6379 is being picked up.

@SiRumCz good catch! Let’s ensure (if needed) you are connecting to an existing cluster (instead of creating a new one), use ray.cluster_resources().

Thanks for all the information! I’m working to figure out what’s wrong.

@anmyachev I have an update, it seems like the problem is not from dot, but the problem might come from the network. I have removed dot operation and the code will still fail with seg fault:

import modin.pandas as pd
import numpy as np

n_features = 20000
n_samples = 20000
features = [f'f{i+1}' for i in range(n_features)]
samples = [f's{i+1}' for i in range(n_samples)]

df = pd.DataFrame(np.random.rand(n_samples, n_features), columns=features, index=samples).round(2)
mat_df = pd.DataFrame(np.random.rand(n_features, n_features), columns=features, index=features).round(2)
res_df = df.apply(lambda x,m: m.iloc[0], axis=1, m=mat_df) # this will still get seg fault

Error message:

>>> df.apply(lambda x,m: m.head(1), axis=1, m=mat_df)
UserWarning: Ray Client is attempting to retrieve a 2.99 GiB object over the network, which may be slow. Consider serializing the object to a file and using S3 or rsync instead.
Segmentation fault (core dumped)

The problem seems to be during or after the Client requested for the object over the network.

Update: While it works fine on simple apply func, it will give me segfault If I include the large matrix df in the apply function (even if not use it):

>>> df.apply(lambda x: x+1)
          f1    f2    f3    f4    f5    f6    f7    f8    f9   f10   f11   f12   f13   f14  ...  f19987  f19988  f19989  f19990  f19991  f19992  f19993  f19994  f19995  f19996  f19997  f19998  f19999  f20000
s1      1.87  1.05  1.88  1.67  1.84  1.12  1.58  1.68  1.33  1.81  1.21  1.33  1.14  1.93  ...    1.89    1.96    1.40    1.08    1.27    1.13    1.81    1.77    1.78    1.56    1.39    1.90    1.15    1.59
...      ...   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...   ...  ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...     ...
s20000  1.28  1.24  1.66  1.27  1.47  1.07  1.06  1.24  1.84  1.57  1.90  1.36  1.01  1.08  ...    1.25    1.44    1.34    1.78    1.94    1.73    1.52    1.60    1.87    1.47    1.07    1.43    1.69    1.47

[20000 rows x 20000 columns]
>>> df.apply(lambda x, m: x+1, m=mat_df)
UserWarning: Ray Client is attempting to retrieve a 2.99 GiB object over the network, which may be slow. Consider serializing the object to a file and using S3 or rsync instead.
Segmentation fault (core dumped)

The matrix parameter in the apply func doesn’t need to be a dataframe, I tried with numpy array and it has the same problem, my guess is that the param data cannot be too large to trigger some mechanism, otherwise I will get seg fault.