modin: dfsql tests failing on Windows/MacOS after the last modin update

I was able to reproduce hanging behavior locally (Reproducibility is not 100%).

Environment:

conda env create -f environment-dev.yml
set MODIN_CPUS=4
set MODIN_ENGINE=ray
pytest modin\pandas\test\test_io.py::TestCsv::test_hanging_behavior --verbose -s

Simplified reproducer (that should be added to TestCsv class):

def test_hanging_behavior(self):
    for i in range(16):
        #print("to_csv")
        pd.DataFrame([1, 2, 3, 4]).to_csv("initial-data.csv", index=False)
        #print("read_csv")
        df = pd.read_csv("initial-data.csv")
        #print("isnull, all, axis=1")
        df.index[df.isnull().all(axis=1)].values.tolist()
        #print("isnull, all, axis=0")
        df.columns[df.isnull().all(axis=0)].values.tolist()

Logs:

...\modin>pytest modin\pandas\test\test_io.py::TestCsv::test_hanging_behavior --verbose -s
=============================================== test session starts ===============================================
platform win32 -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- ...\Miniconda3\envs\modin\python.exe
cachedir: .pytest_cache
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: ...\modin, configfile: setup.cfg
plugins: benchmark-3.4.1, cov-2.11.0, forked-1.3.0, xdist-2.3.0
collected 1 item

modin/pandas/test/test_io.py::TestCsv::test_just_test to_csv
read_csv
(pid=20528) Windows fatal exception: access violation
(pid=20528)
isnull, all, axis=1
isnull, all, axis=0
to_csv
read_csv
isnull, all, axis=1
(pid=15432) Windows fatal exception: access violation
(pid=15432)
isnull, all, axis=0
to_csv
read_csv
isnull, all, axis=1
(pid=23128) Windows fatal exception: access violation
(pid=23128)
isnull, all, axis=0
to_csv
read_csv
isnull, all, axis=1
(pid=3412) Windows fatal exception: access violation
(pid=3412)
isnull, all, axis=0
to_csv
read_csv
(pid=20984) Windows fatal exception: access violation
(pid=20984)
isnull, all, axis=1
isnull, all, axis=0
to_csv
read_csv
(pid=7096) Windows fatal exception: access violation
(pid=7096)
isnull, all, axis=1
isnull, all, axis=0
to_csv
read_csv
isnull, all, axis=1
(pid=23464) Windows fatal exception: access violation
(pid=23464)
isnull, all, axis=0
to_csv
read_csv
(pid=12504) Windows fatal exception: access violation
(pid=12504)
isnull, all, axis=1
isnull, all, axis=0
to_csv
read_csv
isnull, all, axis=1
(pid=11500) Windows fatal exception: access violation
(pid=11500) 
isnull, all, axis=0
to_csv
read_csv
(pid=19948) Windows fatal exception: access violation
(pid=19948)
isnull, all, axis=1
isnull, all, axis=0
to_csv
read_csv
isnull, all, axis=1
(pid=20848) Windows fatal exception: access violation
(pid=20848) 
isnull, all, axis=0
to_csv
2021-08-25 20:03:18,393 WARNING worker.py:1189 -- The actor or task with ID c6cf2fddfe5e7c90b398e5da6a4450ee63f746a18d1ec44e cannot be scheduled right now. It requires {CPU: 1.000000} for placement, but this node only has remaining 
{4.000000/4.000000 CPU, 13.969839 GiB/13.969839 GiB memory, 13.969839 GiB/13.969839 GiB object_store_memory, 1.000000/1.000000 node:10.147.230.30}
. In total there are 1 pending tasks and 0 pending actors on this node. This is likely due to all cluster resources 
being claimed by actors. To resolve the issue, consider creating fewer actors or increase the resources available to this Ray cluster. You can ignore this message if this Ray cluster is expected to auto-scale or if you specified a runtime_env for this task or actor because it takes time to install.

@rkooo567 did this clarify anything?

Finally the issue turned out to be Modin related. Ray hanged when I was trying to write a Modin DataFrame to disk while using Ray. It seems there was some kind of a deadlock, but I am still not sure.

For now I resolved it by using Pandas to write to disk, after which the issue was gone: https://github.com/mindsdb/dfsql/pull/19/files#diff-287da181ac34dcb8710924d3be04f46fac4c8b26c7de303766af97d571d1b969R26

Still, it’s wroth investigating why that was happening.