NVTabular: [QST] Incompatible CUDA Version

I’m using the following container merlin-pytorch:22.12 When I try to execute sample workflow for Transformers4Rec, I’m getting the following errors:

I’m assuming UserWarning is caused by “cat_feats = [‘product_id’] >> nvt.ops.Categorify(start_index=1) >> TagAsItemID()”

What is the most recent way of tagging features?

Is the CUDA version mismatch caused by incompatible versions? I’ve created this ticket on Merlin repo.

` /usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: ‘item’>, <Tags.ID: ‘id’>]. warnings.warn( Failed to transform operator <nvtabular.ops.list_slice.ListSlice object at 0x7fae1e32e3d0> Traceback (most recent call last): File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 2705, in add_ptx driver.cuLinkAddData(self.handle, enums.CU_JIT_INPUT_PTX, File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 320, in safe_cuda_api_call self._check_ctypes_error(fname, retcode) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 388, in _check_ctypes_error raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [222] Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File “/usr/local/lib/python3.8/dist-packages/merlin/dag/executors.py”, line 170, in _transform_data output_data = node.op.transform(selection, input_data) File “/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py”, line 101, in inner result = func(*args, **kwargs) File “/usr/local/lib/python3.8/dist-packages/nvtabular/ops/list_slice.py”, line 111, in transform _calculate_row_sizes[blocks, threads]( File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 491, in call return self.dispatcher.call(args, self.griddim, self.blockdim, File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 625, in call kernel = _dispatcher.Dispatcher._cuda_call(self, *args) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 633, in _compile_for_args return self.compile(tuple(argtypes)) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 796, in compile kernel.bind() File “/usr/local/lib/python3.8/dist-packages/numba/cuda/dispatcher.py”, line 178, in bind self._codelibrary.get_cufunc() File “/usr/local/lib/python3.8/dist-packages/numba/cuda/codegen.py”, line 208, in get_cufunc cubin = self.get_cubin(cc=device.compute_capability) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/codegen.py”, line 181, in get_cubin linker.add_ptx(ptx.encode()) File “/usr/local/lib/python3.8/dist-packages/numba/cuda/cudadrv/driver.py”, line 2708, in add_ptx raise LinkerError(“%s\n%s” % (e, self.error_log)) numba.cuda.cudadrv.driver.LinkerError: [222] Call to cuLinkAddData results in UNKNOWN_CUDA_ERROR ptxas application ptx input, line 9; fatal : Unsupported .version 7.8; current version is ‘7.6’ `

About this issue

  • Original URL
  • State: closed
  • Created a year ago
  • Comments: 38 (20 by maintainers)

Most upvoted comments

@rnyak That solved the issue!

image

@ldane I don’t see an error in your last post? Does it have to do with the dtype of the column?

Yes, I’m showing the dtype of the column. If you can try to load that parquet file with NVTabular, you will come across the TypeError.

@ldane I can read your parquet file with cudf.read_parquet('test.parquet') and I can see that the feed_date dtype as datetime64[s]. Please see the toy example below with your test.parquet dataset.

image

but cannot read this parquet file with pandas. I get TypeError: data type 'dbdate' not understood error with pandas as well…

Are you on 22.12? Could you check if you have db-dtypes installed?

I’ve tried both NVTabular as well as dask_cudf. I’m getting TypeError on both cases. If we can’t solve it, I can look into possible work around as you suggested. image

image