mamba: Error when trying to use Mamba2

Traceback (most recent call last):
  File "test_mambav2.py", line 6, in <module>
    from mamba_ssm import Mamba
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/__init__.py", line 5, in <module>
    from mamba_ssm.modules.mamba2 import Mamba2
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/modules/mamba2.py", line 23, in <module>
    from mamba_ssm.distributed.tensor_parallel import ColumnParallelLinear, RowParallelLinear
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/distributed/tensor_parallel.py", line 14, in <module>
    from src.distributed.distributed_utils import (
ModuleNotFoundError: No module named 'src

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 2
  • Comments: 25

Most upvoted comments

Idk i haven’t tested anything not multiple of 512. You can also not use the conv1d package (uninstalling it should mean the mamba code uses torch nn.Conv1d).

Hi @tridao ,

Thanks for your swift reply. I got the following error without using the conv1d package (can be solved by installing the conv1d package ). Any comments are highly appreciated.

Traceback (most recent call last):
  File "/home/jma/Documents/mamba/debug_mamba2.py", line 12, in <module>
    y = model(x)
  File "/home/jma/anaconda3/envs/mamba2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jma/anaconda3/envs/mamba2/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jma/Documents/mamba/mamba_ssm/modules/mamba2.py", line 176, in forward
    out = mamba_split_conv1d_scan_combined(
  File "/home/jma/Documents/mamba/mamba_ssm/ops/triton/ssd_combined.py", line 908, in mamba_split_conv1d_scan_combined
    return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate)
  File "/home/jma/anaconda3/envs/mamba2/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/jma/anaconda3/envs/mamba2/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 115, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/home/jma/Documents/mamba/mamba_ssm/ops/triton/ssd_combined.py", line 757, in forward
    causal_conv1d_cuda.causal_conv1d_fwd(rearrange(xBC, "b s d -> b d s"),
AttributeError: 'NoneType' object has no attribute 'causal_conv1d_fwd'

@tyshiwo1

I solved it with a little trick:

Assuming you are using the function mamba_chunk_scan_combined in file ssd_combined.py.

First, add this into the beginning of the file to replace the original implementation of shape in pytorch.

if True:
    class tTensor(torch.Tensor):
        @property
        def shape(self):
            shape = super().shape
            return tuple([int(s) for s in shape])
    to_ttensor = lambda *args: tuple([tTensor(x) for x in args]) if len(args) > 1 else tTensor(args[0])

Second, add to_ttensor() to everywhere that is related to the error. For example, in my cases (only using the function mamba_chunk_scan_combined), I need to add x, dt, A, B, C = to_ttensor(x, dt, A, B, C) before line return MambaChunkScanCombinedFn.apply(x, dt, A, B, C, chunk_size, D, z, dt_bias, initial_states, seq_idx, dt_softplus, dt_limit, return_final_states) in mamba_chunk_scan_combined; add dt = to_ttensor(dt) before line states = _chunk_state_fwd(B, x, dt, dA_cumsum, seq_idx=seq_idx, states_in_fp32=True) in _mamba_chunk_scan_combined_fwd; change states, final_states = _state_passing_fwd(rearrange(states, "... p n -> ... (p n)"),... to states, final_states = _state_passing_fwd(to_ttensor(rearrange(states, "... p n -> ... (p n)")), ... in _mamba_chunk_scan_combined_fwd.

And after done all these, the error disappears.

Your filename is wrongly named as k_activation.py when it is importing k_activations. However, I am still facing issue after fixing that. The code works with Mamba v1.

Traceback (most recent call last):
  File "test_mambav2.py", line 266, in <module>
    y = mamba(x.unsqueeze(0))
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/modules/mamba2.py", line 176, in forward
    out = mamba_split_conv1d_scan_combined(
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 908, in mamba_split_conv1d_scan_combined
    return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate)
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/torch/autograd/function.py", line 553, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 115, in decorate_fwd
    return fwd(*args, **kwargs)
  File "/home/test/miniconda3/envs/mamba/lib/python3.8/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 757, in forward
    causal_conv1d_cuda.causal_conv1d_fwd(rearrange(xBC, "b s d -> b d s"),
RuntimeError: causal_conv1d with channel last layout requires strides (x.stride(0) and x.stride(2)) to be multiples of 8

thanks i’ve updated