mamba: ImportError causal_conv1d_cuda.cpython-310-x86_64-linux-gnu.so undefined symbol

I am encountering a strange error, while using Mamba with torch 2.1.1+cu118 on Linux. In another environment with torch 1.13.1+cu116 the same code works fine.

ImportError: /home/user1/.conda/envs/2prod/lib/python3.10/site-packages/causal_conv1d_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda20CUDACachingAllocator12recordStreamERKNS_7DataPtrENS0_10CUDAStreamE

Package Version


absl-py 1.4.0 aiohttp 3.9.1 aiosignal 1.3.1 annotated-types 0.6.0 antlr4-python3-runtime 4.9.3 anyio 3.7.1 appdirs 1.4.4 arrow 1.2.3 async-timeout 4.0.3 attrs 23.1.0 backoff 2.2.1 beautifulsoup4 4.12.2 binaryornot 0.4.4 biopython 1.81 blessed 1.19.1 boto3 1.26.106 botocore 1.29.106 Brotli 1.1.0 brotlipy 0.7.0 build 1.0.3 CacheControl 0.13.1 cachetools 5.3.0 causal-conv1d 1.0.0 certifi 2023.11.17 cffi 1.16.0 chardet 5.1.0 charset-normalizer 3.3.2 cleo 2.1.0 click 8.1.3 cmake 3.25.0 colorama 0.4.6 contourpy 1.0.7 cookiecutter 2.1.1 crashtest 0.4.1 croniter 1.4.1 cryptography 41.0.7 cycler 0.11.0 dataclasses 0.8 datasets 2.8.0 dateutils 0.6.12 deepdiff 6.7.1 deepspeed 0.12.4 dill 0.3.6 distlib 0.3.7 docker-pycreds 0.4.0 dulwich 0.21.7 einops 0.6.0 exceptiongroup 1.2.0 fastapi 0.104.1 fastjsonschema 2.19.0 filelock 3.13.1 fonttools 4.38.0 frozenlist 1.4.0 fsspec 2023.12.1 gitdb 4.0.10 GitPython 3.1.30 gmpy2 2.1.2 google-auth 2.17.1 google-auth-oauthlib 1.0.0 GPUtil 1.4.0 grpcio 1.53.0 h11 0.14.0 hjson 3.1.0 huggingface-hub 0.16.4 hydra-core 1.3.1 idna 3.6 importlib-metadata 7.0.0 inquirer 3.1.4 installer 0.7.0 itsdangerous 2.1.2 jaraco.classes 3.3.0 jeepney 0.8.0 Jinja2 3.1.2 jinja2-time 0.2.0 jmespath 1.0.1 joblib 1.2.0 keopscore 2.1.1 keyring 24.3.0 kiwisolver 1.4.4 lightning 2.1.2 lightning-cloud 0.5.57 lightning-utilities 0.10.0 mamba-ssm 1.0.1 Markdown 3.4.3 markdown-it-py 2.1.0 MarkupSafe 2.1.3 matplotlib 3.6.3 mdurl 0.1.2 more-itertools 10.1.0 mpmath 1.3.0 msgpack 1.0.7 multidict 6.0.4 multiprocess 0.70.14 munkres 1.1.4 networkx 3.2.1 ninja 1.11.1.1 numpy 1.26.2 oauthlib 3.2.2 omegaconf 2.3.0 opt-einsum 3.3.0 ordered-set 4.1.0 orjson 3.9.10 packaging 23.2 pandas 1.5.3 pathtools 0.1.2 patsy 0.5.3 pexpect 4.8.0 Pillow 9.4.0 pip 23.3.1 pkginfo 1.9.6 platformdirs 3.11.0 poetry 1.7.1 poetry-core 1.8.1 poetry-plugin-export 1.6.0 prettytable 3.9.0 prettyTables 1.1.5 protobuf 3.20.3 psutil 5.9.4 ptyprocess 0.7.0 py-cpuinfo 9.0.0 pyahocorasick 2.0.0 pyarrow 10.0.1 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybedtools 0.9.0 pybind11 2.10.3 pycparser 2.21 pydantic 2.1.1 pydantic_core 2.4.0 Pygments 2.14.0 PyJWT 2.8.0 pykeops 2.1.1 pynvml 11.5.0 pyOpenSSL 23.3.0 pyparsing 3.0.9 pyproject_hooks 1.0.0 pysam 0.20.0 PySocks 1.7.1 python-dateutil 2.8.2 python-editor 1.0.4 python-multipart 0.0.6 python-slugify 8.0.1 pytorch-lightning 2.1.1 pytz 2022.7.1 PyYAML 6.0.1 rapidfuzz 3.5.2 readchar 4.0.5.dev0 regex 2022.10.31 requests 2.31.0 requests-oauthlib 1.3.1 requests-toolbelt 1.0.0 responses 0.18.0 rich 13.2.0 rsa 4.9 s3transfer 0.6.0 sacremoses 0.0.53 safetensors 0.3.3 scikit-learn 1.2.2 scipy 1.10.0 seaborn 0.13.0 SecretStorage 3.3.3 sentencepiece 0.1.97 sentry-sdk 1.14.0 seqeval 1.2.2 setproctitle 1.3.2 setuptools 68.2.2 shellingham 1.5.4 six 1.16.0 sklearn 0.0.post2 smmap 5.0.0 sniffio 1.3.0 soupsieve 2.5 starlette 0.27.0 starsessions 1.3.0 statsmodels 0.13.5 sympy 1.12 tensorboard 2.12.1 tensorboard-data-server 0.7.0 tensorboard-plugin-wit 1.8.1 tensorboardX 2.6 text-unidecode 1.3 threadpoolctl 3.1.0 tokenizers 0.13.3 tomli 2.0.1 tomlkit 0.12.3 torch 2.1.1+cu118 torchaudio 2.1.1+cu118 torchmetrics 1.2.1 torchvision 0.16.1+cu118 tqdm 4.66.1 traitlets 5.14.0 transformers 4.33.3 triton 2.1.0 trove-classifiers 2023.11.29 types-python-dateutil 2.8.19.14 typing_extensions 4.8.0 tzdata 2023.3 unicodedata2 15.1.0 urllib3 1.26.0 uvicorn 0.24.0.post1 virtualenv 20.25.0 wandb 0.13.9 wcwidth 0.2.12 websocket-client 1.7.0 websockets 12.0 Werkzeug 2.2.3 wheel 0.42.0 xxhash 3.2.0 yarl 1.9.3 zipp 3.17.0

I tried setting up the environment several times, and it would be great if I could use torch > 2.0.

Thanks for having a look!

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 4
  • Comments: 15

Most upvoted comments

Hi, this seems very related to this issue: https://github.com/state-spaces/mamba/issues/169

Maybe also here, try to pip uninstall mamba-ssm and/or causal-conv1d and pip install them again making sure to prevent using the pip cache by adding the

--no-cache-dir

argument.

I have been successfully run. Environment follows: cuda 11.8 python 3.10.13 pytorch 2.1.1 causal_conv1d 1.1.1 mamba-ssm 1.2.0.post1

pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
pip install causal_conv1d==1.1.1
pip install mamba-ssm==1.2.0.post1

Any solution on this now?

I find torch 2.2 will have this problem, after switch it to torch 2.1, I solve it

I was using the PyTorch 2.3 and it was not working. On the contrary, I downgrade to PyTorch 2.2. it is working currently. and below are the configurations of ‘causal-conv1d’ and 'mamba-ssm ’ causal-conv1d 1.1.3.post1 pypi_0 pypi mamba-ssm 1.2.0.post1 pypi_0 pypi