cudf: [BUG] is_monotonic crashes with memory-pool enabled
Describe the bug
Using cudf.set_allocator(pool=True) is causing a segmentation fault in certain calls to is_monotonic (see simple reproducer below). The failure seems to occur within libcudf’s is_sorted.
Steps/Code to reproduce bug
import cudf
import numpy as np
cudf.set_allocator(pool=True)
df = cudf.DataFrame({"a": np.arange(10)})
df.a.is_monotonic
Output: Segmentation fault
The script finishes as expected when using cudf.set_allocator(pool=False)
Note that I originally discovered this problem while modifying the dask_cudf join implementation to use that of main-line Dask. In that case, I sometimes get the same Segmentation fault as above, but sometimes get a bit more information:
...
File "/home/nfs/rzamora/workspace/cudf-dask-devel/dask-bugfixes/dask/dataframe/methods.py", line 89, in boundary_slice
if kind == "loc" and not df.index.is_monotonic:
File "/home/nfs/rzamora/workspace/cudf-dask-devel/cudf-bugfixes/python/cudf/cudf/core/index.py", line 797, in is_monotonic
return self._values.is_monotonic
File "/home/nfs/rzamora/workspace/cudf-dask-devel/cudf-bugfixes/python/cudf/cudf/core/column/column.py", line 778, in is_monotonic
return self.is_monotonic_increasing
File "/home/nfs/rzamora/workspace/cudf-dask-devel/cudf-bugfixes/python/cudf/cudf/core/column/numerical.py", line 404, in is_monotonic_increasing
[self]
File "cudf/_lib/issorted.pyx", line 17, in cudf._lib.issorted.issorted
File "cudf/_lib/issorted.pyx", line 41, in cudf._lib.issorted.issorted
RuntimeError: after reduction step 2: invalid resource handle
Expected behavior
I wouldn’t expect a legal is_monotonic call (like in the reproducer above) to crash.
Environment overview (please complete the following information)
- Environment location: Bare-metal
- Method of cuDF install: from source
Environment details
Click here to see environment details
**git*** commit d8ba85a1ca0d114ffcffd39ef58739961ad598d5 (HEAD) Merge: b40a6d3 c4a72f6 Author: Jake Hemstad <jhemstad@nvidia.com> Date: Thu Oct 24 23:27:40 2019 -0500 Merge pull request #3065 from kaatish/fea-select-concat-table [REVIEW] Select and Concat for cudf exp table **git submodules*** b165e1fb11eeea64ccf95053e40f2424312599cc thirdparty/cub (v1.7.1) 63f644be44201467e3938d59ed9d89cc8725c35d thirdparty/jitify (remotes/origin/feature/api_v2_v0.10) 5654f2bcc8972d513cf9cedf28510c1b6613aeeb thirdparty/libcudacxx (0.8.1-82-g5654f2b) +a4ace1ce72e2e5376c4e3965e632357e7d55e0b6 thirdparty/libcudacxx/libcxx (heads/master) ***OS Information*** DGX_NAME="DGX Server" DGX_PRETTY_NAME="NVIDIA DGX Server" DGX_SWBUILD_DATE="2018-03-20" DGX_SWBUILD_VERSION="3.1.6" DGX_COMMIT_ID="1b0f58ecbf989820ce745a9e4836e1de5eea6cfd" DGX_SERIAL_NUMBER=QTFCOU8310024 DGX_OTA_VERSION="3.1.7" DGX_OTA_DATE="Thu Sep 27 20:07:53 PDT 2018" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS" NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.5 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial Linux dgx16 4.4.0-135-generic #161-Ubuntu SMP Mon Aug 27 10:45:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ***GPU Information*** Fri Oct 25 12:02:51 2019 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 396.44 Driver Version: 396.44 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:06:00.0 Off | 0 | | N/A 41C P0 46W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 Tesla V100-SXM2... On | 00000000:07:00.0 Off | 0 | | N/A 44C P0 46W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 Tesla V100-SXM2... On | 00000000:0A:00.0 Off | 0 | | N/A 44C P0 46W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 Tesla V100-SXM2... On | 00000000:0B:00.0 Off | 0 | | N/A 42C P0 46W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 4 Tesla V100-SXM2... On | 00000000:85:00.0 Off | 0 | | N/A 44C P0 47W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 5 Tesla V100-SXM2... On | 00000000:86:00.0 Off | 0 | | N/A 43C P0 45W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 6 Tesla V100-SXM2... On | 00000000:89:00.0 Off | 0 | | N/A 45C P0 45W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 7 Tesla V100-SXM2... On | 00000000:8A:00.0 Off | 0 | | N/A 41C P0 44W / 300W | 11MiB / 32510MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ***CPU*** Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0-79 Thread(s) per core: 2 Core(s) per socket: 20 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 79 Model name: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz Stepping: 1 CPU MHz: 2211.000 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 4391.73 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 51200K NUMA node0 CPU(s): 0-19,40-59 NUMA node1 CPU(s): 20-39,60-79 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts flush_l1d ***CMake*** /home/nfs/rzamora/miniconda3/envs/cudf_11_dev/bin/cmake cmake version 3.15.4 CMake suite maintained and supported by Kitware (kitware.com/cmake). ***g++*** /usr/bin/g++ g++ (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ***nvcc*** ***Python*** /home/nfs/rzamora/miniconda3/envs/cudf_11_dev/bin/python Python 3.7.3 ***Environment Variables*** PATH : /home/nfs/rzamora/.vscode-server/bin/6ab598523be7a800d7f3eb4d92d7ab9a66069390/bin:/home/nfs/rzamora/bin:/home/nfs/rzamora/.local/bin:/home/nfs/rzamora/miniconda3/envs/cudf_11_dev/bin:/home/nfs/rzamora/miniconda3/condabin:/home/nfs/rzamora/.vscode-server/bin/6ab598523be7a800d7f3eb4d92d7ab9a66069390/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games LD_LIBRARY_PATH : NUMBAPRO_NVVM : /usr/local/cuda-9.2/nvvm/lib64/libnvvm.so NUMBAPRO_LIBDEVICE : /usr/local/cuda-9.2/nvvm/libdevice CONDA_PREFIX : /home/nfs/rzamora/miniconda3/envs/cudf_11_dev PYTHON_PATH : ***conda packages*** /home/nfs/rzamora/miniconda3/condabin/conda # packages in environment at /home/nfs/rzamora/miniconda3/envs/cudf_11_dev: # # Name Version Build Channel _libgcc_mutex 0.1 main aiohttp 3.6.2 pypi_0 pypi alabaster 0.7.12 py_0 conda-forge appdirs 1.4.3 py_1 conda-forge arrow-cpp 0.15.0 py37h090bef1_1 conda-forge asn1crypto 1.2.0 py37_0 conda-forge aspy.yaml 1.3.0 py_0 conda-forge async-timeout 3.0.1 pypi_0 pypi atomicwrites 1.3.0 py_0 conda-forge attrs 19.3.0 py_0 conda-forge babel 2.7.0 py_0 conda-forge backcall 0.1.0 py_0 conda-forge black 19.3b0 py_0 conda-forge bleach 3.1.0 py_0 conda-forge bokeh 1.3.4 py37_0 conda-forge boost-cpp 1.70.0 h8e57a91_2 conda-forge brotli 1.0.7 he1b5a44_1000 conda-forge bzip2 1.0.8 h516909a_1 conda-forge c-ares 1.15.0 h516909a_1001 conda-forge ca-certificates 2019.10.16 0 cached-property 1.5.1 py_0 conda-forge certifi 2019.9.11 py37_0 cffi 1.13.1 py37h8022711_0 conda-forge cfgv 2.0.1 py_0 conda-forge chardet 3.0.4 py37_1003 conda-forge click 7.0 py_0 conda-forge cloudpickle 1.2.2 py_0 conda-forge cmake 3.15.4 hf94ab9c_0 conda-forge cmake_setuptools 0.1.3 py_0 rapidsai-nightly commonmark 0.9.1 py_0 conda-forge cryptography 2.7 py37h72c5cf5_0 conda-forge cudatoolkit 9.2 0 nvidia cudf 0.11.0a0+1730.g539416f.dirty dev_0 <develop> cudnn 7.6.0 cuda9.2_0 nvidia cupy 6.3.0 py37h6adefed_0 rapidsai curl 7.65.3 hf8cf82a_0 conda-forge cython 0.29.13 py37he1b5a44_0 conda-forge cytoolz 0.10.0 py37h516909a_0 conda-forge dask 2.6.0+7.g94c6b16 dev_0 <develop> dask-cuda 0.10.0a1+9.g3f1a4f2 dev_0 <develop> dask-cudf 0.11.0a0+1730.g539416f.dirty dev_0 <develop> decorator 4.4.0 py_0 conda-forge defusedxml 0.6.0 py_0 conda-forge distributed 2.6.0+7.g876bca0 pypi_0 pypi dlpack 0.2 he1b5a44_1 conda-forge docutils 0.15.2 py37_0 conda-forge double-conversion 3.1.5 he1b5a44_1 conda-forge editdistance 0.5.3 py37hf484d3e_0 conda-forge entrypoints 0.3 py37_1000 conda-forge expat 2.2.5 he1b5a44_1004 conda-forge fastavro 0.22.5 py37h516909a_0 conda-forge fastrlock 0.4 py37hf484d3e_1000 conda-forge flake8 3.7.8 py37_1 conda-forge flatbuffers 1.11.0 he1b5a44_0 conda-forge freetype 2.10.0 he983fc9_1 conda-forge fsspec 0.5.2 py_0 conda-forge future 0.18.1 py37_0 conda-forge gflags 2.2.2 he1b5a44_1001 conda-forge glog 0.4.0 he1b5a44_1 conda-forge gmp 6.1.2 hf484d3e_1000 conda-forge grpc-cpp 1.23.0 h18db393_0 conda-forge heapdict 1.0.1 py_0 conda-forge hypothesis 4.41.3 py37_0 conda-forge icu 64.2 he1b5a44_1 conda-forge identify 1.4.7 py_0 conda-forge idna 2.8 py37_1000 conda-forge imagesize 1.1.0 py_0 conda-forge importlib_metadata 0.23 py37_0 conda-forge ipykernel 5.1.3 py37h5ca1d4c_0 conda-forge ipython 7.8.0 py37h5ca1d4c_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge isort 4.3.21 py37_0 conda-forge jedi 0.15.1 py37_0 conda-forge jinja2 2.10.3 py_0 conda-forge jpeg 9c h14c3975_1001 conda-forge json5 0.8.5 py_0 jsonschema 3.1.1 py37_0 conda-forge jupyter-server-proxy 1.1.0 pypi_0 pypi jupyter_client 5.3.3 py37_1 conda-forge jupyter_core 4.5.0 py_0 conda-forge jupyterlab 1.1.4 pyhf63ae98_0 jupyterlab-nvdashboard 0.1.11 pypi_0 pypi jupyterlab_server 1.0.6 py_0 krb5 1.16.3 h05b26f9_1001 conda-forge libblas 3.8.0 14_openblas conda-forge libcblas 3.8.0 14_openblas conda-forge libcurl 7.65.3 hda55be3_0 conda-forge libedit 3.1.20170329 hf8c457e_1001 conda-forge libevent 2.1.10 h72c5cf5_0 conda-forge libffi 3.2.1 he1b5a44_1006 conda-forge libgcc-ng 9.1.0 hdf63c60_0 libgfortran-ng 7.3.0 hdf63c60_2 conda-forge liblapack 3.8.0 14_openblas conda-forge libopenblas 0.3.7 h6e990d7_2 conda-forge libpng 1.6.37 hed695b0_0 conda-forge libprotobuf 3.8.0 h8b12597_0 conda-forge librmm 0.11.0a191024 cuda9.2_123 rapidsai-nightly libsodium 1.0.17 h516909a_0 conda-forge libssh2 1.8.2 h22169c7_2 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 libtiff 4.0.10 hfc65ed5_1004 conda-forge libuv 1.33.1 h516909a_0 conda-forge llvmlite 0.29.0 py37hfd453ef_1 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.8.3 he1b5a44_1001 conda-forge markdown 3.0.1 pypi_0 pypi markupsafe 1.1.1 py37h14c3975_0 conda-forge mccabe 0.6.1 py_1 conda-forge mistune 0.8.4 py37h14c3975_1000 conda-forge more-itertools 7.2.0 py_0 conda-forge msgpack-python 0.6.2 py37hc9558a2_0 conda-forge multidict 4.5.2 pypi_0 pypi nbconvert 5.6.0 py37_1 conda-forge nbformat 4.4.0 py_1 conda-forge nbsphinx 0.4.3 py_0 conda-forge nccl 2.4.6.1 cuda9.2_0 nvidia ncurses 6.1 hf484d3e_1002 conda-forge nodeenv 1.3.3 py_0 conda-forge nodejs 10.13.0 he6710b0_0 notebook 6.0.1 py37_0 conda-forge npm 0.1.1 pypi_0 pypi numba 0.45.1 py37hb3f55d8_0 conda-forge numpy 1.17.3 py37h95a1406_0 conda-forge numpydoc 0.9.1 py_0 conda-forge nvstrings-cudaunknown 0.0.0.dev0 pypi_0 pypi olefile 0.46 py_0 conda-forge openssl 1.1.1d h7b6447c_3 optional-django 0.1.0 pypi_0 pypi packaging 19.2 py_0 conda-forge pandas 0.24.2 py37hb3f55d8_0 conda-forge pandoc 1.19.2 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parquet-cpp 1.5.1 2 conda-forge parso 0.5.1 py_0 conda-forge partd 1.0.0 py_0 conda-forge pexpect 4.7.0 py37_0 conda-forge pickleshare 0.7.5 py37_1000 conda-forge pillow 6.2.1 py37h6b7be26_0 conda-forge pip 19.3.1 py37_0 conda-forge pluggy 0.13.0 py37_0 conda-forge pre_commit 1.18.1 py37_0 conda-forge prometheus_client 0.7.1 py_0 conda-forge prompt_toolkit 2.0.10 py_0 conda-forge psutil 5.6.3 py37h516909a_0 conda-forge ptyprocess 0.6.0 py_1001 conda-forge py 1.8.0 py_0 conda-forge pyarrow 0.15.0 py37h8b68381_1 conda-forge pycodestyle 2.5.0 py_0 conda-forge pycparser 2.19 py37_1 conda-forge pyflakes 2.1.1 py_0 conda-forge pygments 2.4.2 py_0 conda-forge pynvml 8.0.3 pypi_0 pypi pyopenssl 19.0.0 py37_0 conda-forge pyparsing 2.4.2 py_0 conda-forge pyrsistent 0.15.4 py37h516909a_0 conda-forge pysocks 1.7.1 py37_0 conda-forge pytest 5.2.1 py37_0 conda-forge python 3.7.3 h33d41f4_1 conda-forge python-dateutil 2.8.0 py_0 conda-forge pytz 2019.3 py_0 conda-forge pyyaml 5.1.2 py37h516909a_0 conda-forge pyzmq 18.1.0 py37h1768529_0 conda-forge rapidjson 1.1.0 he1b5a44_1002 conda-forge re2 2019.09.01 he1b5a44_0 conda-forge readline 8.0 hf8c457e_0 conda-forge recommonmark 0.6.0 py_0 conda-forge requests 2.22.0 py37_1 conda-forge rhash 1.3.6 h14c3975_1001 conda-forge rmm 0.11.0a191024 py37_123 rapidsai-nightly send2trash 1.5.0 py_0 conda-forge setuptools 41.4.0 py37_0 conda-forge simpervisor 0.3 pypi_0 pypi six 1.12.0 py37_1000 conda-forge snakeviz 2.0.1 pypi_0 pypi snappy 1.1.7 he1b5a44_1002 conda-forge snowballstemmer 2.0.0 py_0 conda-forge sortedcontainers 2.1.0 py_0 conda-forge sphinx 2.2.0 py_0 conda-forge sphinx-markdown-tables 0.0.10 pypi_0 pypi sphinx_rtd_theme 0.4.3 py_0 conda-forge sphinxcontrib-applehelp 1.0.1 py_0 conda-forge sphinxcontrib-devhelp 1.0.1 py_0 conda-forge sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.2 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge sphinxcontrib-websupport 1.1.2 py_0 conda-forge sqlite 3.30.1 hcee41ef_0 conda-forge streamz 0.5.2 pypi_0 pypi tblib 1.4.0 py_0 conda-forge terminado 0.8.2 py37_0 conda-forge testpath 0.4.2 py_1001 conda-forge thrift-cpp 0.12.0 hf3afdfd_1004 conda-forge tk 8.6.9 hed695b0_1003 conda-forge toml 0.10.0 py_0 conda-forge toolz 0.10.0 py_0 conda-forge tornado 6.0.3 py37h516909a_0 conda-forge traitlets 4.3.3 py37_0 conda-forge uriparser 0.9.3 he1b5a44_1 conda-forge urllib3 1.25.6 py37_0 conda-forge virtualenv 16.7.5 py_0 conda-forge wcwidth 0.1.7 py_1 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.33.6 py37_0 conda-forge xz 5.2.4 h14c3975_1001 conda-forge yaml 0.1.7 h14c3975_1001 conda-forge yarl 1.3.0 pypi_0 pypi zeromq 4.3.2 he1b5a44_2 conda-forge zict 1.0.0 py_0 conda-forge zipp 0.6.0 py_0 conda-forge zlib 1.2.11 h516909a_1006 conda-forge zstd 1.4.3 h3b9ef0a_0 conda-forge
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 21 (21 by maintainers)
Nevermind. I already see what the problem is.
That works!
#3277 is resolving my join bug as well 😃