vllm: TypeError: PaddedGatherOp.forward() takes 6 positional arguments but 7 were given

I installed from latest main, installed stk, megablocks, latest flash_attn, transformers etc… And got following error:

  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/vllm/engine/ray_utils.py", line 32, in execute_method
    return executor(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/vllm/worker/worker.py", line 88, in profile_num_available_blocks
    self.model_runner.profile_run()
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 321, in profile_run
    self.execute_model(seqs, kv_caches)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 279, in execute_model
    hidden_states = self.model(
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/vllm/model_executor/models/mixtral.py", line 488, in forward
    hidden_states = layer(
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/vllm/model_executor/models/mixtral.py", line 439, in forward
    r = self.block_sparse_moe(self.ffn_norm(h))
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/vllm/model_executor/models/mixtral.py", line 353, in forward
    x = ops.padded_gather(x, indices, bin_ids, bins, padded_bins,
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/ubuntu/mambaforge/lib/python3.10/site-packages/stk/backend/autocast.py", line 28, in decorate_fwd
    return fwd(*args, **kwargs)
TypeError: PaddedGatherOp.forward() takes 6 positional arguments but 7 were given

_Originally posted by @draganjovanovich in https://github.com/vllm-project/vllm/issues/2011#issuecomment-1849769275_

About this issue

Original URL
State: closed
Created 7 months ago
Comments: 16

Most upvoted comments

This issue is because pip install megablocks installs megablocks==0.2.0 which is outdated. Please install the latest megablocks by pip install git+https://github.com/stanford-futuredata/megablocks.git@v0.5.0

WoosukKwon on Dec 11, 2023

You can install with --ignore-dependencies. The only packages MegaBlocks needs to run Mixtral is stanford-stk and triton. If you give me 10m this will be fixed and pip install megablocks will be all you need for Mixtral 😃

tgale96 on Dec 11, 2023

Ok, thanks, will try now. btw, I also would love to get some bucks(“megabucks”) via pip install lol

draganjovanovich on Dec 11, 2023