arrow-adbc: adbc_driver_manager.OperationalError: UNKNOWN: [Snowflake] arrow/ipc: unknown error while reading: cannot allocate memory

Hello,

Was trying the adbc python snow driver and I see something odd . Simple function

# Query via adbc
def query_via_adbc_arrow(conn_adbc, query, return_type='arrow'):
    print(f"Enters query_via_adbc_arrow  ")
    start_time = datetime.datetime.now()
    cursor = conn_adbc.cursor()
    cursor.execute(query)
    # Return pandas or arrow
    if return_type == 'arrow':
        df = cursor.fetch_arrow_table()
        num_rows = df.num_rows
    else:
        df = cursor.fetch_df() 

        #df = cursor.fetch_arrow_table().to_pandas()
        num_rows = df.shape[0]
    cursor.close()
    end_time = datetime.datetime.now()
    execution_time = (end_time - start_time).total_seconds()
    print(f" Got {num_rows} rows  in {return_type} . The execution time is {execution_time} secs ")
    print(f"Exits query_via_adbc_arrow ")

Sometimes it works and some times it does not ( the regular snowflake python connector works every time ) When we run adbc part there are weird errors (310) [aborissov@bhsys-data-dev-euw1c-lnx10 python_project2]$ python3 snow_python_adbc.py Enters query_via_adbc_arrow Got 10003143 rows in arrow . The execution time is 1.62057 secs Exits query_via_adbc_arrow Enters query_via_adbc_arrow Traceback (most recent call last): File “/<>/python_project2/snow_python_adbc.py”, line 133, in <module> df = query_via_adbc_arrow(conn_adbc, “select * from private.main_td_limit_orders”, return_type=‘pandas’) File “/bh<>/snow_python_adbc.py”, line 91, in query_via_adbc_arrow df = cursor.fetch_df() # Does not work well with large data running out of memory File “/<>/venvs/310/lib/python3.10/site-packages/adbc_driver_manager/dbapi.py”, line 1050, in fetch_df return self._results.fetch_df() File “<>/venvs/310/lib/python3.10/site-packages/adbc_driver_manager/dbapi.py”, line 1139, in fetch_df return self._reader.read_pandas() File “adbc_driver_manager/_reader.pyx”, line 108, in adbc_driver_manager._reader.AdbcRecordBatchReader.read_pandas File “adbc_driver_manager/_reader.pyx”, line 40, in adbc_driver_manager._reader._AdbcErrorHelper.check_error adbc_driver_manager.OperationalError: UNKNOWN: [Snowflake] arrow/ipc: unknown error while reading: cannot allocate memory (310) [<>python_project2]$ (310) [<> python_project2]$ (310) [<> python_project2]$ (310) [<> python_project2]$ (310) [<> python_project2]$ (310) [<>python_project2]$ python3 snow_python_adbc.py Enters query_via_adbc_arrow Got 10003143 rows in arrow . The execution time is 1.535479 secs Exits query_via_adbc_arrow Enters query_via_adbc_arrow Traceback (most recent call last): File “/<>/python_project2/python_project2/snow_python_adbc.py”, line 133, in <module> df = query_via_adbc_arrow(conn_adbc, “select * from private.main_td_limit_orders”, return_type=‘pandas’) File “/<>/python_project2/python_project2/snow_python_adbc.py”, line 91, in query_via_adbc_arrow df = cursor.fetch_df() # Does not work well with large data running out of memory File “/<>/venvs/310/lib/python3.10/site-packages/adbc_driver_manager/dbapi.py”, line 1050, in fetch_df return self._results.fetch_df() File “<>/venvs/310/lib/python3.10/site-packages/adbc_driver_manager/dbapi.py”, line 1139, in fetch_df return self._reader.read_pandas() File “adbc_driver_manager/_reader.pyx”, line 108, in adbc_driver_manager._reader.AdbcRecordBatchReader.read_pandas File “adbc_driver_manager/_reader.pyx”, line 40, in adbc_driver_manager._reader._AdbcErrorHelper.check_error adbc_driver_manager.OperationalError: UNKNOWN: [Snowflake] arrow/ipc: unknown error while reading: cannot allocate memory (310) [<> python_project2]$ (310) [<> python_project2]$ (310) [<> python_project2]$ (310) [<> python_project2]$ (310) [<> python_project2]$ python3 snow_python_adbc.py Enters query_via_adbc_arrow Got 10003143 rows in arrow . The execution time is 1.560988 secs Exits query_via_adbc_arrow Enters query_via_adbc_arrow Got 10003143 rows in pandas . The execution time is 2.706668 secs Exits query_via_adbc_arrow (310) [<> python_project2]$

Has anyone experienced that ? Thanks

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 1
  • Comments: 37 (24 by maintainers)

Commits related to this issue

Most upvoted comments

Oops, I think 0.11.0 didn’t actually incorporate the fix (the release happened right before the fix got in).

There are nightly wheels here, do you want to try them? (Download the artifact “python-amd64-manylinux2014”, extract it, and manually pip install path/to/wheel.whl) https://github.com/apache/arrow-adbc/actions/runs/8792621756

first argument somewhere between 800000 and 1200000, second argument is 1

So I managed to reproduce the “cannot allocate memory” error locally and interactive. With a bit of manipulation I was able to get stack traces and look closer at the situation. The “cannot allocate memory” error is being directly returned by the call to calloc in Mallocator.Allocate. The request for the allocation that it fails on seems to be somewhere between 800KB and a bit over 1MB, so it’s not failing due to an invalid (0 or negative) size for allocation. The failure is also happening while i still have more than 10 - 12 GB of RAM available to me.

I’m only able to reproduce this issue via the python script in the repo provided by @bascheibler, I haven’t been able to create a reproducer with just Go.

So after getting it to run with the updated build, I now got the failure to allocate memory error that was originally filed. I haven’t been able to reproduce that with pure Go yet. So I’m not calling this solved just yet until I can determine the cause of that allocation error.

It is based on nanoarrow, but is unrelated to anything in ADBC.

I’ll try to use the updated repo scripts and see if i can reproduce it myself and figure out what I’m missing

sigh As with any complex system, it came down to a race condition. @shollington-rbi @bascheibler @aab200 could any of you try pulling down the branch from the linked PR and verifying it fixes the problem for you?

By setting the prefetch concurrency option to “1” i was able to reproduce the close of nil channel error with a pure go mainprog. I’m going to dig in to try to determine a solution

We are also seeing this issue sporadically too when executing a fetch_arrow_table() on a AWS Lambda Python script when connecting to Snowflake. @bascheibler - were you able to find a combination of parameter configuration that minimizes the frqeuncy?

Here’s a repo with both the dataset and the scripts to reproduce the issue: https://github.com/bascheibler/arrow-adbc-issue-1283

I’ve created only 3 test samples with hashed data off of some of the tables that failed. Unfortunately, I wasn’t able to test this new dataset because the adbc_ingest function is either not committing or its payload is too large (please see my comment in the main.py file). The tables that get created have 0 rows in Snowflake. However, please feel free to use these parquet files as work best for you.

Please let me know if this data is enough to reproduce the issues. We may need to force a longer loop with repeated tables in order to get an error. Not sure how it behaves with a shorter list of tables.

maybe timeouts or something on the snowflake side? yea, this would be easiest if we could have a dataset that can reproduce this. I’ll try tomorrow to see if i can generate one unless @bascheibler is able to provide something roughly equivalent to what he’s using (I assume the actual data you’re using is likely not able to be released publicly) that is able to trigger the errors.

@bascheibler could you provide any kind of equivalent dataset via parquet/csv/otherwise that I could easily load into my own snowflake instance to attempt to reproduce this?

Ok. The Go traceback indicates an actual bug that needs fixing.

The dataset described doesn’t sound big enough to be worthy of causing memory issues, either. (Unless it’s very highly compressed while in Snowflake.) So something seems off.