vllm: [Bug]: When running gemma2 7b, an error is reported [rank0]: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP)` Set up according to the prompts: os.environ['VLLM_ATTENTION_BACKEND'] = 'FLASHINFER' print("Environment variable set for VLLM_ATTENTION_BACKEND:", os.getenv('VLLM_ATTENTION_BACKEND'))

Your current environment

When running gemma2 7b, an error is reported [rank0]: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP) Set up according to the prompts: os.environ[‘VLLM_ATTENTION_BACKEND’] = ‘FLASHINFER’ print(“Environment variable set for VLLM_ATTENTION_BACKEND:”, os.getenv(‘VLLM_ATTENTION_BACKEND’))

🐛 Describe the bug

When running gemma2 7b, an error is reported [rank0]: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16BF, lda, b, CUDA_R_16BF, ldb, &fbeta, c, CUDA_R_16BF, ldc, compute_type, CUBLAS_GEMM_DEFAULT_TENSOR_OP) Set up according to the prompts: os.environ[‘VLLM_ATTENTION_BACKEND’] = ‘FLASHINFER’ print(“Environment variable set for VLLM_ATTENTION_BACKEND:”, os.getenv(‘VLLM_ATTENTION_BACKEND’))

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 18 (1 by maintainers)

Most upvoted comments

One should install FlashInfer manually @orderer0001

But, currently it can only use one GPU.

update: got Segmentation fault (core dumped) after inferenced about 50 requests