datasets: Fatal error condition occurred in aws-c-io

Describe the bug

Fatal error when using the library

Steps to reproduce the bug

from datasets import load_dataset
dataset = load_dataset('wikiann', 'en')

Expected results

No fatal errors

Actual results

Fatal error condition occurred in D:\bld\aws-c-io_1633633258269\work\source\event_loop.c:74: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application

Environment info

datasets version: 1.15.2.dev0
Platform: Windows-10-10.0.22504-SP0
Python version: 3.8.12
PyArrow version: 6.0.0

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 2
Comments: 28 (4 by maintainers)

Commits related to this issue

Downgrade pyarrow. see https://github.com/huggingface/datasets/issues/3310#issuecomment-1247390774 — committed to thesofakillers/infoshare by thesofakillers a year ago

Most upvoted comments

Downgrading pyarrow to 6.0.1 solves the issue for me.

pip install pyarrow==6.0.1

+26

boru-roylu on Sep 14, 2022

I also get this issue, It appears after my script has finished running. I get the following error message

Fatal error condition occurred in /home/conda/feedstock_root/build_artifacts/aws-c-io_1637179816120/work/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_backtrace_print+0x59) [0x2aabe0479579]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_fatal_assert+0x48) [0x2aabe04696c8]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0x13ad3) [0x2aabe0624ad3]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aabe047b60d]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0x113ca) [0x2aabe06223ca]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aabe047b60d]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-crt-cpp.so(_ZN3Aws3Crt2Io15ClientBootstrapD1Ev+0x3a) [0x2aabe041cf5a]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../.././libaws-cpp-sdk-core.so(+0x5f570) [0x2aabe00eb570]
/lib64/libc.so.6(+0x39ce9) [0x2aaaab835ce9]
/lib64/libc.so.6(+0x39d37) [0x2aaaab835d37]
/lib64/libc.so.6(__libc_start_main+0xfc) [0x2aaaab81e55c]
python(+0x1c721d) [0x55555571b21d]
Aborted

I don’t get this issue when running my code in a container, and it seems more relevant to PyArrow but thought a more complete stack trace might be helpful to someone

Any updates for your issue because I’m getting the same one

gitbooo on Aug 26, 2022

I also get this issue, It appears after my script has finished running. I get the following error message

Fatal error condition occurred in /home/conda/feedstock_root/build_artifacts/aws-c-io_1637179816120/work/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_backtrace_print+0x59) [0x2aabe0479579]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_fatal_assert+0x48) [0x2aabe04696c8]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0x13ad3) [0x2aabe0624ad3]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aabe047b60d]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0x113ca) [0x2aabe06223ca]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aabe047b60d]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../../././libaws-crt-cpp.so(_ZN3Aws3Crt2Io15ClientBootstrapD1Ev+0x3a) [0x2aabe041cf5a]
/home/user_name/conda_envs/env_name/lib/python3.7/site-packages/pyarrow/../../.././libaws-cpp-sdk-core.so(+0x5f570) [0x2aabe00eb570]
/lib64/libc.so.6(+0x39ce9) [0x2aaaab835ce9]
/lib64/libc.so.6(+0x39d37) [0x2aaaab835d37]
/lib64/libc.so.6(__libc_start_main+0xfc) [0x2aaaab81e55c]
python(+0x1c721d) [0x55555571b21d]
Aborted

I don’t get this issue when running my code in a container, and it seems more relevant to PyArrow but thought a more complete stack trace might be helpful to someone

CallumMcMahon on Dec 6, 2021

pyarrow 10.0.1 was just released in conda-forge, which is the first release where we’re building against aws-sdk-cpp 1.9.* again after more than a year. Since we cannot test the failure reported here on our infra, I’d be very grateful if someone could verify that the problem does or doesn’t reappear. 🙃

conda install -c conda-forge pyarrow=10

h-vetinari on Dec 4, 2022

I also encountered the same problem, but I made an error in the multi gpu training environment on Linux, and the single gpu training environment will not make an error. i use accelerate package to do multi gpu training.

leileilin on Aug 25, 2022

For pip people, I confirmed that installing the nightly version of pyarrow also solves this by: pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ --prefer-binary --pre pyarrow --upgrade. (See https://arrow.apache.org/docs/python/install.html#installing-nightly-packages) Any version after https://github.com/apache/arrow/pull/14157 would work fine.

kngwyu on Oct 26, 2022

There is also a discussion here https://issues.apache.org/jira/browse/ARROW-15141 where it is suggested for conda users to use an older version of aws-sdk-cpp: aws-sdk-cpp=1.8.186

lhoestq on Sep 6, 2022