MMseqs2: libmarv initialization error during GPU-accelerated search against ColabFold databases using Docker

Summary

When running GPU-accelerated search using the mmseqs2 docker image against the colabfold_envdb_202108 or uniref30_2302_db databases I see an error saying

CUDA error: initialization error : /opt/build/lib/libmarv/src/marv.cu, line 85
Error: Prefilter died

That particular line is involved in getting the CUDA device count, so maybe it has something to do with seeing the GPUs on the instanc?

Environment

  • AWS g5.8xlarge EC2 instance
    • AMD EPYC 7R32 processor
    • 32x vCPU
    • 128 GiB Memory
    • 1x NVIDIA A10G accelerator (24 GiB VRAM)
  • Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.5.1 (Amazon Linux 2023) 20250107
    • 6.1.119-129.201.amzn2023.x86_64 Linux distro
    • CUDA Version: 12.6
    • CUDA Driver Version: 560.35.03
    • PyTorch 2.5.1
  • Container ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12
    • Pulled Wednesday, January 22, 2025

Steps to reproduce

mkdir data
wget https://www.rcsb.org/fasta/entry/1UTN -O data/1utn.fasta
wget https://wwwuser.gwdg.de/~compbiol/colabfold/colabfold_envdb_202108.tar.gz
tar -xzvf colabfold_envdb_202108.tar.gz -C data
rm colabfold_envdb_202108.tar.gz
docker pull ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12
docker run -it --rm --gpus all -v "$(pwd)/data:/home/data" ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12 tsv2exprofiledb "/home/data/colabfold_envdb_202108" "/home/data/targetDB" --gpu 1
docker run -it --rm --gpus all -v "$(pwd)/data:/home/data" ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12 createdb "/home/data/1utn.fasta" "/home/data/queryDB"
docker run -it --rm --gpus all -v "$(pwd)/data:/home/data" ghcr.io/soedinglab/mmseqs2:17-b804f-cuda12 search "/home/data/queryDB" "/home/data/targetDB" "/home/data/result" "/home/data/tmp" --num-iterations 3 --db-load-mode 2 -a -e 0.1 --max-seqs 10000 --gpu 1 --prefilter-mode 1

Things I’ve tried

✅ SUCCESS: Run search with --gpu 0 (i.e. turn off GPU acceleration) ⛔ ERROR: Set the CUDA_VISIBLE_DEVICES="0" environment variable ⛔ ERROR: Use the ghcr.io/soedinglab/mmseqs2:latest-cuda12 container ⛔ ERROR: Clone the GitHub repo and build the Dockerfile ⛔ ERROR: Modify the Dockerfile to COPY /opt/build/lib/libmarv from the build stage ⛔ ERROR: Modify the Dockerfile to use the precompiled binary ⛔ ERROR: Run using a g5.12xlarge (4x A10G GPUs instead of 1x)

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 19 (7 by maintainers)

Most upvoted comments

Reopening as we need to keep investigating. MMseqs2-GPU should work even on Turing (albeit rather slowly). Ampere and newer should all work fine.

@brianloyal would you mind sharing the AMI (I didn’t find it on AWS cuda 12.4) and how you started the docker ultimately?

Sure thing - this was the winner https://aws.amazon.com/releasenotes/aws-deep-learning-ami-gpu-pytorch-2-5-ubuntu-22-04/

One more update. I switched over to a Ubuntu AMI running CUDA 12.4 on a A10G and it worked. I’ve seen some reports in the past of CUDA version inconsistencies when compiling on a A100 vs A10G using Amazon Linux so it’s probably something to do with that. However, since (A) it works on a g5 running Ubuntu, and (B) it works on a g6 and p4 running Amazon Linux, I’m satisfied. No edits necessary to the wiki.

Does the following script work on the A10G instance:

wget https://mmseqs.com/latest/mmseqs-linux-gpu.tar.gz
tar xzvf mmseqs-linux-gpu.tar.gz
wget https://raw.githubusercontent.com/soedinglab/MMseqs2/refs/heads/master/examples/QUERY.fasta
./mmseqs/bin/mmseqs easy-search QUERY.fasta QUERY.fasta res tmp --gpu 1

If not, can you try this script on an L4 or L40s based instance? We did most of our testing on those.

Is CUDA_VISIBLE_DEVICES set to some odd value?