espnet: Top 20 slowest tests

I found running test is really slower now!

• master: 30:45
• v 0.9.0: 10:38
• v 0.8.0: NA
• v 0.7.0: 13:45
• v 0.6.0: 5:09

in CircleCI’s ubuntu-18 image (running out of free credits!)

I think unit tests should be fast because test-driven dev requires that property (while the integration tests can be slow). Here are the top 20 slowest test files:

• 0 (‘test/test_beam_search.py’, 536.879999999997) -> 1.91 (#2462)
• 1 (‘test/test_batch_beam_search.py’, 289.2799999999999) -> 1.45 (#2462)
• 2 (‘test/test_e2e_asr_mulenc.py’, 216.11999999999998) -> 8.86 (#2480)
• 3 (‘test/espnet2/enh/nets/test_beamformer_net.py’, 212.21000000002525) -> 12.76s (#2489)
• 4 (‘test/espnet2/tts/test_transformer.py’, 69.75999999999985) -> 5.30 (#2461)
• 5 (‘test/test_e2e_st_transformer.py’, 49.809999999999995) -> 8.0 (#2464)
• 6 (‘test/test_asr_init.py’, 42.18) -> 5.4 (#2444)
• 7 (‘test/test_e2e_asr_sa_transducer.py’, 38.73000000000001) -> (#98dof55: 43.5) -> 5.89 (#2444)
• 8 (‘test/test_e2e_asr_transformer.py’, 34.8) -> 5.0 (#2464)
• 9 (‘test/test_e2e_asr_conformer.py’, 32.07) -> 5.0 (#2464)
• 10 (‘test/test_recog.py’, 22.89) -> 5.18 (#2486)
• 11 (‘test/test_e2e_asr.py’, 21.729999999999997) -> 9.79s (#2488)
• 12 (‘test/espnet2/tts/test_tacotron2.py’, 18.9999999999998) -> 2.15 (#2461)
• 13 (‘test/espnet2/tts/test_fastspeech2.py’, 17.110000000000078) -> 2.19 (#2461)
• 14 (‘test/espnet2/enh/nets/test_tasnet.py’, 16.99999999999999) -> 7.01 (#2491)
• 15 (‘test/test_e2e_mt_transformer.py’, 15.910000000000002) -> 5.0 (#2464)
• 16 (‘test/test_e2e_asr_maskctc.py’, 12.639999999999999) -> 5.07 (#2493)
• 17 (‘test/espnet2/tts/test_fastspeech.py’, 8.749999999999961) -> 3.10 (#2461)
• 18 (‘test/espnet2/asr/decoder/test_transformer_decoder.py’, 8.039999999999996)
• 19 (‘test/espnet2/lm/test_seq_rnn_lm.py’, 5.039999999999998)

top20 total 0:27:49.950000 (test python: total 30:45)

I summarized the benchmark per file because I like well-organized smaller test files, and easy to start PRs per files. Ideally, if we can run these tests within ~1 sec, unit tests only take 3m in total.

Appendix: script to calc top 20

# wget https://circleci.com/api/v1.1/project/github/espnet/espnet/20951/output/105/0?file=true&allocation-id=5f5bdcac8763316b07dced1c-0-build%2F4AD5A548 -O0.txt
import datetime
import sys
import collections

d = collections.defaultdict(int)
begin = False
for line in open("0.txt", "r"):
    if not begin:
        if "slowest durations" in line:
            begin = True
        continue
    if begin:
        if line.strip() == "":
            break
    ls = line.split()
    d[ls[2].split("::")[0]] += float(ls[0][:-1])

total = 0
for i, l in enumerate(sorted(d.items(), key=lambda kv: -kv[1])[:20]):
    print(i, l)
    total += l[1]
print(datetime.timedelta(seconds=total))

0.txt

tips for faster testing

• replace file IOs with mock functions (file IO gets very slow sometimes in CI)
• replace importlib with real import statements
• use the smallest parameters (e.g., batch_size=2)
• remove unnecessary (i.e., no contribution to coverage) pytest.parameterized cases