DeepSpeed: Fail to install sparse attention and cpu-adam

When I used install.sh, I received the following warnings:

`[WARNING] fused lamb is NOT installed.

[WARNING] transformer kernels are NOT installed.

[WARNING] sparse attention is NOT installed.

[WARNING] cpu-adam (used by ZeRO-offload) is NOT installed.`

After changing the wheel installation to pure pip installation, the fused lamb and transformer kernel can be installed. But I still fail to install the sparse attention and cpu-adam.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 17 (8 by maintainers)

Commits related to this issue

Triton kernels and BERT inference using triton in float16 (#459) * Add residual_add triton op * add support of gptj style models to triton residual_add kernel * fix the residual_add tests * ... — committed to microsoft/DeepSpeed by stephen-youn a year ago

Most upvoted comments

And for installing Sparse Attention, you have the following dependencies:

llvm: sudo apt-get install llvm-9-dev
cmake: sudo apt-get install cmake
triton>=0.2.2: this will be handled by the installer script, if there was any issue, please install it through pip

Then you need to set the install flag: DS_BUILD_SPARSE_ATTN=1 (for sparse attention) DS_BUILD_CPU_ADAM=1 (for cpu adam)

and finally run ./install.sh

Please update us on the result.

arashashari on Oct 6, 2020

I failed to import triton after pip install it. So I have to compile it from source with several other dependencies. After these, I can install sparse attention successfully.

szhengac on Oct 6, 2020

Thanks for help guys! I’ve managed to install cpu adam, but for now i can’t install llvm-9-dev with apt due to environment limitations. Will try to build it locally from source

ollmer on Oct 6, 2020

@arashashari Thanks for fixing my typo on DS_BUILD_CPU_ADAM

tjruwase on Oct 6, 2020