vllm: [Bug]: When running gemma2 7b, an error is reported [rank0]: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)` Set up according to the prompts: os.environ['VLLM_ATTENTION_BACKEND'] = 'FLASHINFER' print("Environment variable set for VLLM_ATTENTION_BACKEND:", os.getenv('VLLM_ATTENTION_BACKEND'))
Your current environment
When running gemma2 7b, an error is reported [rank0]: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
Set up according to the prompts:
os.environ[‘VLLM_ATTENTION_BACKEND’] = ‘FLASHINFER’
print(“Environment variable set for VLLM_ATTENTION_BACKEND:”, os.getenv(‘VLLM_ATTENTION_BACKEND’))
🐛 Describe the bug
When running gemma2 7b, an error is reported [rank0]: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)
Set up according to the prompts:
os.environ[‘VLLM_ATTENTION_BACKEND’] = ‘FLASHINFER’
print(“Environment variable set for VLLM_ATTENTION_BACKEND:”, os.getenv(‘VLLM_ATTENTION_BACKEND’))
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (1 by maintainers)
One should install FlashInfer manually @orderer0001
But, currently it can only use one GPU.
update: got
Segmentation fault (core dumped)after inferenced about 50 requests