MMseqs2: libmarv initialization error during GPU-accelerated search against ColabFold databases using Docker
Summary
When running GPU-accelerated search using the mmseqs2 docker image against the colabfold_envdb_202108 or uniref30_2302_db databases I see an error saying
CUDA error: initialization error : /opt/build/lib/libmarv/src/marv.cu, line 85
Error: Prefilter died
That particular line is involved in getting the CUDA device count, so maybe it has something to do with seeing the GPUs on the instanc?
Environment
- AWS g5.8xlarge EC2 instance
- AMD EPYC 7R32 processor
- 32x vCPU
- 128 GiB Memory
- 1x NVIDIA A10G accelerator (24 GiB VRAM)
- Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.5.1 (Amazon Linux 2023) 20250107
- 6.1.119-129.201.amzn2023.x86_64 Linux distro
- CUDA Version: 12.6
- CUDA Driver Version: 560.35.03
- PyTorch 2.5.1
- Container ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12
- Pulled Wednesday, January 22, 2025
Steps to reproduce
mkdir data
wget https://www.rcsb.org/fasta/entry/1UTN -O data/1utn.fasta
wget https://wwwuser.gwdg.de/~compbiol/colabfold/colabfold_envdb_202108.tar.gz
tar -xzvf colabfold_envdb_202108.tar.gz -C data
rm colabfold_envdb_202108.tar.gz
docker pull ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12
docker run -it --rm --gpus all -v "$(pwd)/data:/home/data" ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12 tsv2exprofiledb "/home/data/colabfold_envdb_202108" "/home/data/targetDB" --gpu 1
docker run -it --rm --gpus all -v "$(pwd)/data:/home/data" ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12 createdb "/home/data/1utn.fasta" "/home/data/queryDB"
docker run -it --rm --gpus all -v "$(pwd)/data:/home/data" ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12 search "/home/data/queryDB" "/home/data/targetDB" "/home/data/result" "/home/data/tmp" --num-iterations 3 --db-load-mode 2 -a -e 0.1 --max-seqs 10000 --gpu 1 --prefilter-mode 1
Things I’ve tried
✅ SUCCESS: Run search with --gpu 0 (i.e. turn off GPU acceleration)
⛔ ERROR: Set the CUDA_VISIBLE_DEVICES="0" environment variable
⛔ ERROR: Use the ghcr.io/soedinglab/mmseqs2:latest-cuda12 container
⛔ ERROR: Clone the GitHub repo and build the Dockerfile
⛔ ERROR: Modify the Dockerfile to COPY /opt/build/lib/libmarv from the build stage
⛔ ERROR: Modify the Dockerfile to use the precompiled binary
⛔ ERROR: Run using a g5.12xlarge (4x A10G GPUs instead of 1x)
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 19 (7 by maintainers)
Reopening as we need to keep investigating. MMseqs2-GPU should work even on Turing (albeit rather slowly). Ampere and newer should all work fine.
Sure thing - this was the winner https://aws.amazon.com/releasenotes/aws-deep-learning-ami-gpu-pytorch-2-5-ubuntu-22-04/
One more update. I switched over to a Ubuntu AMI running CUDA 12.4 on a A10G and it worked. I’ve seen some reports in the past of CUDA version inconsistencies when compiling on a A100 vs A10G using Amazon Linux so it’s probably something to do with that. However, since (A) it works on a g5 running Ubuntu, and (B) it works on a g6 and p4 running Amazon Linux, I’m satisfied. No edits necessary to the wiki.
Does the following script work on the A10G instance:
If not, can you try this script on an L4 or L40s based instance? We did most of our testing on those.
Is CUDA_VISIBLE_DEVICES set to some odd value?