cuml: [BUG] RuntimeError: radix_sort: failed on 2nd step: cudaErrorInvalidValue: invalid argument
Hi all,
I am using the latest version of the rapids.ai docker. 21.06 and in Juptyer notebook this code works with no issue:
from cuml.common.sparsefuncs import csr_row_normalize_l2
def efficient_csr_cosine_similarity(query, tfidf_matrix, matrix_normalized=False):
query = csr_row_normalize_l2(query, inplace=False)
if not matrix_normalized:
tfidf_matrix = csr_row_normalize_l2(tfidf_matrix, inplace=False)
return tfidf_matrix.dot(query.T)
def document_search(text_df, query, vectorizer, tfidf_matrix, top_n=3):
query_vec = vectorizer.transform(Series([query]))
similarities = efficient_csr_cosine_similarity(query_vec, tfidf_matrix, matrix_normalized=True)
similarities = similarities.todense().reshape(-1)
best_idx = similarities.argsort()[-top_n:][::-1]
pp = cudf.DataFrame({
'text': text_df['workout'].iloc[best_idx],
'similarity': similarities[best_idx]
})
return pp
document_search(wod_df, 'time', vec, tfidf_matrix)
But when I run it with straight python on the same docker for a dash application, I get this runtime error:
Traceback (most recent call last): File “/home/robomike/code/coolstuff_gpu/Merlin/cool/app/work/app.py”, line 118, in displayStuff main_df = document_search(cool_df, ‘for time’, vec, tfidf_matrix) File “/home/robomike/code/coolstuff_gpu/Merlin/cool/app/work/app.py”, line 43, in document_search best_idx = similarities.argsort()[-top_n:][::-1] File “cupy/_core/core.pyx”, line 715, in cupy._core.core.ndarray.argsort
File “cupy/_core/core.pyx”, line 732, in cupy._core.core.ndarray.argsort
File “cupy/_core/_routines_sorting.pyx”, line 86, in cupy._core._routines_sorting._ndarray_argsort
File “cupy/cuda/thrust.pyx”, line 117, in cupy.cuda.thrust.argsort
RuntimeError: radix_sort: failed on 2nd step: cudaErrorInvalidValue: invalid argument Attached is the py file that causes the error.app2.py (1.4 KB)
I really don’t understand what is going wrong here. Can someone please help me?
Update: This problem also persists on rapidsai/rapidsai-nightly:21.08-cuda11.0-runtime-ubuntu20.04-py3.7
Update 2: I’ve tried this by removing a GPU, the issue still persists. I have GTX Titan and GTX 1080.
Steps/Code to reproduce bug call function document_search in python
Expected behavior should work with no problems.
Environment details (please complete the following information): docker: rapidsai/rapidsai-nightly:21.08-cuda11.0-runtime-ubuntu20.04-py3.7 docker: rapidsai/rapidsai 21.06 stable
Additional context Original code written by Nvidia link is here
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 1
- Comments: 33 (10 by maintainers)
same issue with GTX1080 for SVM
File ~/anaconda3/envs/rapids/lib/python3.8/site-packages/cupy/manipulation/add_remove.py:179, in unique(ar, return_index, return_inverse, return_counts, axis) 177 aux = ar[perm] 178 else: –> 179 ar.sort() 180 aux = ar 181 mask = cupy.empty(aux.shape, dtype=cupy.bool)
File cupy/_core/core.pyx:729, in cupy._core.core.ndarray.sort()
File cupy/_core/core.pyx:747, in cupy._core.core.ndarray.sort()
File cupy/_core/_routines_sorting.pyx:43, in cupy._core._routines_sorting._ndarray_sort()
File cupy/cuda/thrust.pyx:75, in cupy.cuda.thrust.sort()
RuntimeError: radix_sort: failed on 2nd step: cudaErrorInvalidValue: invalid argument
hi @Nanthini10 Here is the docker environment where the error still happened:
import dash_html_components as htmlwithfrom dash import htmlimport dash_html_components as html ^C(rapids) root@bdf4ab8a676b:/rapids/notebooks/nc/Merlin/wodmatic/app/workouts# sh print_env.shClick here to see environment details
Hi @dantegd got just the thing for you. workout_exp.csv
And here is the code for python:
`import os import cudf from blazingsql import BlazingContext from cudf import Series from cudf import DataFrame
from cuml.feature_extraction.text import TfidfVectorizer
from cuml.common.sparsefuncs import csr_row_normalize_l2
def efficient_csr_cosine_similarity(query, tfidf_matrix, matrix_normalized=False): query = csr_row_normalize_l2(query, inplace=False) if not matrix_normalized: tfidf_matrix = csr_row_normalize_l2(tfidf_matrix, inplace=False)
return tfidf_matrix.dot(query.T)
def document_search(text_df, query, vectorizer, tfidf_matrix, top_n=10): query_vec = vectorizer.transform(Series([query])) similarities = efficient_csr_cosine_similarity(query_vec, tfidf_matrix, matrix_normalized=True) similarities = similarities.todense().reshape(-1) best_idx = similarities.argsort()[-top_n:][::-1] pp = cudf.DataFrame({ ‘text’: text_df[‘workout’].iloc[best_idx], ‘similarity’: similarities[best_idx] }) return pp
if name == ‘main’: os.environ[“CUDA_VISIBLE_DEVICES”]=‘0’ wod_df = cudf.read_csv(‘./data/workout_exp.csv’) vec = TfidfVectorizer(stop_words=‘english’) workouts = Series(wod_df[‘workout’]) tfidf_matrix = vec.fit_transform(workouts) tfidf_matrix.shape print(tfidf_matrix.shape) print(document_search(wod_df, ‘time’, vec, tfidf_matrix))`
@taureandyernv this seems to resolve the problem on Docker. For some reason, it doesn’t work in my Conda environment. But at least I can work with the docker. Thank you. 😃