NVTabular: [QST] Incompatible CUDA Version
I’m using the following container merlin-pytorch:22.12 When I try to execute sample workflow for Transformers4Rec, I’m getting the following errors:
I’m assuming UserWarning is caused by “cat_feats = [‘product_id’] >> nvt.ops.Categorify(start_index=1) >> TagAsItemID()”
What is the most recent way of tagging features?
Is the CUDA version mismatch caused by incompatible versions? I’ve created this ticket on Merlin repo.
` /usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: ‘item’>, <Tags.ID: ‘id’>]. warnings.warn( Failed to transform operator <nvtabular.ops.list_slice.ListSlice object at 0x7fae1e32e3d0> Traceback (most recent call last): File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 2705, in add_ptx driver.cuLinkAddData(self.handle, enums.CU_JIT_INPUT_PTX, File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 320, in safe_cuda_api_call self._check_ctypes_error(fname, retcode) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 388, in _check_ctypes_error raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [222] Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File “/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py”, line 170, in _transform_data output_data = node.op.transform(selection, input_data) File “/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py”, line 101, in inner result = func(*args, **kwargs) File “/usr/local/lib/python3.8/dist-packages/nvtabular/ops/list_slice.py”, line 111, in transform _calculate_row_sizes[blocks, threads]( File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 491, in call return self.dispatcher.call(args, self.griddim, self.blockdim, File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 625, in call kernel = _dispatcher.Dispatcher._cuda_call(self, *args) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 633, in _compile_for_args return self.compile(tuple(argtypes)) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 796, in compile kernel.bind() File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 178, in bind self._codelibrary.get_cufunc() File “/usr/local/lib/python3.8/dist-packages/numba/cuda/codegen.py”, line 208, in get_cufunc cubin = self.get_cubin(cc=device.compute_capability) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/codegen.py”, line 181, in get_cubin linker.add_ptx(ptx.encode()) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 2708, in add_ptx raise LinkerError(“%s\n%s” % (e, self.error_log)) numba.cuda.cudadrv.driver.LinkerError: [222] Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR ptxas application ptx input, line 9; fatal : Unsupported .version 7.8; current version is ‘7.6’ `
About this issue
- Original URL
- State: closed
- Created a year ago
- Comments: 38 (20 by maintainers)
@rnyak That solved the issue!
Are you on 22.12? Could you check if you have db-dtypes installed?
I’ve tried both NVTabular as well as dask_cudf. I’m getting TypeError on both cases. If we can’t solve it, I can look into possible work around as you suggested.