MMseqs2: mmseqs local runs considerably slower than mmseqs colabfold api regardless of optimizations
Some issues / unexpected behavior when running mmseqs search and other features used in colabfold. But the low speed happens in the mmseqs step, so I think this is the place to ask.
Unexpected observations:
- running mmseqs locally on our cluster is much slower than the responses we get from the colabfold server for equivalent inputs
- mmseqs search speed does not scale at all with added threads or added physical cores, for small inputs (1-2 sequences) - is this expected? Will it only speed up with large input count?
Welcoming any input! Many thanks!
Expected Behavior
Runtime of mmseqs2 for a given input similar to response time from the colabfold server.
Current Behavior
Considerably slower. For a given input (~250 aa) the server gives a full result in 20-30s, paired + unpaired reads (so, including at least 1 search each against uniref30 and colabfold_envdb, plus a number of expandaln / align calls). The local version takes 60-90 seconds to run just the initial mmseqs search against uniref30, when using vmtouch and db load mode 2, and ~300 seconds against colabfold_envdb. Db load mode 3 somewhat slower, db load mode 0 and 1 much slower as expected. Index was created without splits. The mmseqs search step is timed after vmtouch and other preparations are complete. The runtime is independent of either thread count or number of cpus provided. Splitting the target db index 2 4 8 or 16-fold and running in parallel with MPI does not speed it up either (slightly slows down). Placing the entire run directory fast SSD or RAM (/dev/shm) does not change the speed. This happens with the docker version (either avx, sse2 or sse41 versions), and the self-compiled version.
This also happens when a random aminoacid sequence of equivalent length is used as input, so I’m excluding caching on the colabfold api server side as a cause for it to be so fast.
Steps to Reproduce (for bugs)
cat <<'EOF' >mmseq_in.fasta
>UbcH7_Hs_1
MAASRRLMKELEEIRKCGMKNFRNIQVDEANLLTWQGLIVPDNPPYDKGAFRIEINFPAEYPFKPPKITFKTKIYHPNIDEKGQVCLPVISAENWKPATKTDQVIQSLIALVNDPQPEHPLRADLAEEYSKDRKKFCKNAEEFTKKYGEKRPVD
>Ub
MQIFVKTLTGKTITLEVEPSDTIENVKAKIQDKEGIPPDQQRLIFAGKQLEDGRTLSDYNIQKESTLHLVLRLRGG
EOF
mkdir -p out/tmp
mmseqs createdb mmseqs_in.fasta out/qdb
vmtouch -ldw /path/to/database/uniref30_2202_db*idx*
mmseqs search out/qdb /path/to/database/uniref30_2202_db out/res out/tmp --threads 1 --num-iterations 3 --db-load-mode 2 -a -s 8 -e 0.1 --max-seqs 10000
MMseqs Output (for bugs)
attached out_apptainer_run.log
Context
Providing context helps us come up with a solution and improve our documentation for the future.
Your Environment
Include as many relevant details about the environment you experienced the bug in.
- Git commit used (The string after “MMseqs Version:” when you execute MMseqs without any parameters):
- Which MMseqs version was used (Statically-compiled, self-compiled, Homebrew, etc.):
mmseqs2 docker: 7e2840992948ee89dcc336522dc98a74fe0adf00
self-compiled: ed4c55fa21c402e0a9497e5db5cd4b687a75fa02 ed4c55fa21c402e0a9497e5db5cd4b687a75fa02-MPI
-
For self-compiled and Homebrew: Compiler and Cmake versions used and their invocation: cmake version 3.21.0 cmake -DHAVE_MPI=0 -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=. … && make && make install cmake -DHAVE_MPI=1 -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=. … && make && make install
-
Server specifications (especially CPU support for AVX2/SSE and amount of system memory): tried multiple servers, with available memory ranging from 100G to 2000G support for AVX2/SSE all there (see attached cmake output) cmake_out.txt
-
Operating system and version: LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.9.2009 (Core) Release: 7.9.2009 Codename: Core
Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 152 On-line CPU(s) list: 0-151 Thread(s) per core: 2 Core(s) per socket: 19 Socket(s): 4 NUMA node(s): 4 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel® Xeon® Gold 6138 CPU @ 2.00GHz Stepping: 4 CPU MHz: 2000.000 BogoMIPS: 4000.00 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0-37 NUMA node1 CPU(s): 38-75 NUMA node2 CPU(s): 76-113 NUMA node3 CPU(s): 114-151 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology
About this issue
- Original URL
- State: open
- Created 4 years ago
- Comments: 22 (7 by maintainers)
Sorry I was traveling. I don’t see anything out of the ordinary in your screenshot. I still don’t really have an idea whats going on. I repeated the UniRef search with your query on the MSA server:
One thing we do differently since we still have enough spare CPU capacity on the server is enable some limited multithreading and parallel processing.
We use
--threads 2, which will hide quite a bit of overhead in your use-case:And we also start the env-db and template search in parallel to the expand-align steps. (
path.colabfold.parallelstagesparameter in the API server). This will hide quite a bit of latency. However, the initial search is still a lot faster for us despite worse CPU/RAM specs.Can you check if
vmtouchhas actually but everything into RAM?htopshould look something like this: