fairseq: Segmentation fault when training speech_to_text model following instruction in examples/speech_to_text

🐛 Bug

I have followed the README in examples/speech_to_text to reproduce results of ST on MUSTC. But when I start training (after preprocess following the instruction), the system raises segmentation fault error, just after reading dev subset.

Below is part of messages:

2020-10-23 18:09:18 | INFO | fairseq.tasks.speech_to_text | dictionary size (spm_bpe10000_st.txt): 10,000
2020-10-23 18:09:18 | INFO | fairseq.tasks.speech_to_text | pre-tokenizer: {'tokenizer': None}
2020-10-23 18:09:18 | INFO | fairseq.tasks.speech_to_text | tokenizer: {'bpe': 'sentencepiece', 'sentencepiece_model': '/home/ma-user/work/data/mustc-s2t/en-de/spm_bpe10000_st.model'}
2020-10-23 18:09:18 | INFO | fairseq.data.audio.speech_to_text_dataset | SpeechToTextDataset(split="valid_st", n_samples=1388, prepend_tgt_lang_tag=False, shuffle=False, transforms=None)
mustc-test-s2t-cd.sh: line 11: 33140 Segmentation fault      CUDA_VISIBLE_DEVICES=0 python fairseq_cli/train.py ${data_dir} --config-yaml config_st.yaml --train-subset train_st --valid-subset valid_st --save-dir ${model_dir} --num-workers 1 --max-tokens 20000 --task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-update 100000 --arch s2t_transformer_s --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt --warmup-updates 10000 --clip-norm 10.0 --seed 1

To Reproduce

CUDA_VISIBLE_DEVICES=0 python fairseq_cli/train.py ${data_dir} --config-yaml config_st.yaml --train-subset train_st --valid-subset valid_st --save-dir ${model_dir} --num-workers 1 --max-tokens 20000 --task speech_to_text --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-update 100000 --arch s2t_transformer_s --optimizer adam --lr 2e-3 --lr-scheduler inverse_sqrt --warmup-updates 10000 --clip-norm 10.0 --seed 1

Code sample

Expected behavior

Environment

• fairseq Version (e.g., 1.0 or master): 1.0
• PyTorch Version (e.g., 1.0): 1.4.0
• OS (e.g., Linux): Linux
• How you installed fairseq (pip, source): source
• Build command you used (if compiling from source):
• Python version: 3.7
• CUDA/cuDNN version:
• GPU models and configuration:
• Any other relevant information:

Additional context