mamba: Can't train mamba2 from scratch with HF Trainer

I’m trying to train mamba2 130m from scratch.

config = Mamba2Config(
            vocab_size=len(tokenizer.vocab),
            n_positions=10,
            n_embd=768,              
            n_layer=12,               
            n_head=12,             
            n_inner=3072, 
        )
        model = Mamba2ForCausalLM(config)
    training_args = TrainingArguments(
    output_dir=args.output_dir,
    logging_dir='./logs',
    gradient_accumulation_steps=1,  
    save_steps=2000,
    max_steps=1500000, 
    evaluation_strategy="steps",
    eval_steps=2000,
    #prediction_loss_only=True,
    logging_strategy="epoch",
    #logging_steps=500,
    learning_rate=1e-4,
    # save_total_limit=999999,
    fp16=True,
    dataloader_num_workers=4,
    per_device_train_batch_size=512,
    per_device_eval_batch_size=512, 
    lr_scheduler_type="constant_with_warmup",
    weight_decay=0.1,
    warmup_steps=2000,
    )
    trainer = Trainer(
        model=model,
        tokenizer=tokenizer,
        args=training_args,
        train_dataset=tokenized_train,
        eval_dataset=tokenized_eval
    )
trainer.train()

Error:

File "/users/PAS2581/kanaka/research/GrokkedTransformersarewang2024/trying_different_archs/mamba/main.py", line 542, in <module>
    main()
  File "/users/PAS2581/kanaka/research/GrokkedTransformersarewang2024/trying_different_archs/mamba/main.py", line 512, in main
    trainer.train()
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/trainer.py", line 1938, in train
    return inner_training_loop(
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/trainer.py", line 2279, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/trainer.py", line 3318, in training_step
    loss = self.compute_loss(model, inputs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/trainer.py", line 3363, in compute_loss
    outputs = model(**inputs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
    return model_forward(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 43, in decorate_autocast
    return func(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/models/mamba2/modeling_mamba2.py", line 1048, in forward
    mamba2_outputs = self.backbone(
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/models/mamba2/modeling_mamba2.py", line 900, in forward
    hidden_states = mixer_block(
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/models/mamba2/modeling_mamba2.py", line 648, in forward
    hidden_states = self.mixer(
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/models/mamba2/modeling_mamba2.py", line 607, in forward
    return self.torch_forward(hidden_states, cache_params, cache_position, attention_mask)
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/models/mamba2/modeling_mamba2.py", line 531, in torch_forward
    L = torch.exp(segment_sum(A))
  File "/users/PAS2581/kanaka/miniconda3/envs/grokk/lib/python3.10/site-packages/transformers/models/mamba2/modeling_mamba2.py", line 105, in segment_sum
    tensor_segsum = torch.cumsum(input_tensor, dim=-2)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.15 GiB of which 3.70 GiB is free. Including non-PyTorch memory, this process has 75.44 GiB memory in use. Of the allocated memory 74.80 GiB is allocated by PyTorch, and 139.31 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.

where,

GPU 0: NVIDIA A100-SXM4-80GB
  GPU Utilization: 0%
  Memory Utilization: 0%
  Total Memory: 81920.00 MB
  Free Memory: 52246.00 MB
  Used Memory: 29674.00 MB

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 26 (13 by maintainers)

Most upvoted comments

Thank you for the help!

Then the second part of my reply should be correct (?), i.e. max len should be 2048 or longer to be good on speed.

Yes.

@Lynnzake At least in hf, you guarantee that inference is avoided 100% and in that case the code opts for the fused path, i.e. a kernel for conv combined with the mamba op. Anything “kernelized” is faster than the base ops one after the other (even if they are one kernel each).

Not sure what you mean with max_len but the mamba2 kernel at least is pretty unoptimized in regards to smaller seq lens. The speed only gets pretty good at lengths 2048<=

The scaling worked because the effective size didn’t change @Jellymoon

You also have to pass the correct number of heads, i.e.:

import torch
from transformers import AutoTokenizer, Mamba2Model, Mamba2Config

hidden_size = 2048
expand = 2
head_dim = 64
# this here is the key to pass too, not very nice to have it passed in the config imo but that's the current state now :(
num_heads = (hidden_size * expand) // head_dim

config = Mamba2Config(
    vocab_size=len(tokenizer.vocab),
    state_size=128,
    num_heads=num_heads,
    head_dim=head_dim,
    num_hidden_layers=4,
    hidden_size=hidden_size,
    expand=expand,
    # I'd suggest using this as it's default for all models in the mamba-ssm repo
    n_groups=1,
)

...

Sure, Thanks. Please give me an update when you did so.

You’re missing either the mamba-ssm package and/or causal-conv1d package: return self.torch_forward(hidden_states, cache_params, cache_position, attention_mask) <-- this is the slow path, unoptimized and using a lot of memory

Hf only opts for the fast path, i.e. the kernels, if both are available (there should’ve been a warning). So consider installing the packages. For the future, these type of questions are more appropriate on the transformers repo 😉