fairseq: hypo.word file missing during MMS ASR inference
❓ Questions and Help
What is your question?
I’m facing the following issue while running the MMS ASR inference script examples/mms/asr/infer/mms_infer.py:
File "/workspace/fairseq/examples/mms/asr/infer/mms_infer.py", line 52, in <module>
process(args)
File "/workspace/fairseq/examples/mms/asr/infer/mms_infer.py", line 44, in process
with open(tmpdir/"hypo.word") as fr:
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/tmpsjatjyxt/hypo.word'
Code
python examples/mms/asr/infer/mms_infer.py --model "/workspace/fairseq/mms1b_fl102.pt" --lang "urd-script_arabic" --audio "/workspace/audio.wav"
What have you tried?
Tried running the ASR on different audios and languages
What’s your environment?
- fairseq Version (e.g., 1.0 or main): main
- PyTorch Version (e.g., 1.0): 2.0.0
- OS (e.g., Linux): Linux
- How you installed fairseq (
pip, source): pip - Build command you used (if compiling from source): N/A
- Python version: 3.10.10
- CUDA/cuDNN version: 11.6
- GPU models and configuration: NVIDIA A6000
- Any other relevant information: N/A
About this issue
- Original URL
- State: open
- Created a year ago
- Reactions: 31
- Comments: 90 (8 by maintainers)
Can someone share a working colab notebook? I think it might help. Facing the same problems as above.
How to Transcribe Audio to text (Google Colab Version)👇
Step 1: Clone the Fairseq Git Repo
Step 2: Install requirements and build
Be patient, takes some minutes
!pip install --editable ./Step 3: Install Tensor Board
!pip install tensorboardXStep 4: Download your preferred model
Un-comment to download any. If you’re not using Google Colab pro then use a smaller model to avoid memory outrage
Step 5: Upload your audio(s)
Create a folder on path ‘/content/audio_samples/’ and upload your .wav audio files that you need to transcribe e.g. ‘/content/audio_samples/small_trim4.wav’ Note: You need to make sure that the audio data you are using has a sample rate of 16000 You can easily do this with FFMPEG like the example below that converts .mp3 file to .wav and fixing the audio sample rate
Step 6: Run Inference and transcribe your audio(s)
Takes some time for long audios
After this you’ll get your preffered transcription I have this Collab Example in my GitHub Repo👉 fairseq_meta_mms_Google_Colab_implementation
hi @audiolion, after installing omegaconf & hydra a new error appeared
Please y’all read the error messages and try to debug yourself.
@dakouan18
you need to install the missing modules, one of them being
omegaconf@altryne you need to print the error output to debug
@shsagnik your hydra install has some issues, and you need to specify a checkpoint directory, it was setup to run on linux where you can make directories off the root (probably in a container) so change
infer_common.yamlGot these errors this time
preparing tmp manifest dir … loading model & running inference … /home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/core/plugins.py:202: UserWarning: Error importing ‘hydra_plugins.hydra_colorlog’. Plugin is incompatible with this Hydra version or buggy. Recommended to uninstall or upgrade plugin. ImportError : cannot import name ‘SearchPathPlugin’ from ‘hydra.plugins’ (/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/plugins/init.py) warnings.warn( Traceback (most recent call last): File “/home/hello_automate_ai/miniconda3/lib/python3.10/pathlib.py”, line 1175, in mkdir self._accessor.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: ‘/checkpoint/hello_automate_ai/INFER/None’
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File “/home/hello_automate_ai/miniconda3/lib/python3.10/pathlib.py”, line 1175, in mkdir self._accessor.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: ‘/checkpoint/hello_automate_ai/INFER’
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File “/home/hello_automate_ai/miniconda3/lib/python3.10/pathlib.py”, line 1175, in mkdir self._accessor.mkdir(self, mode) FileNotFoundError: [Errno 2] No such file or directory: ‘/checkpoint/hello_automate_ai’
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File “/home/hello_automate_ai/fairseqmmstest/fairseq/examples/speech_recognition/new/infer.py”, line 499, in <module> cli_main() File “/home/hello_automate_ai/fairseqmmstest/fairseq/examples/speech_recognition/new/infer.py”, line 495, in cli_main hydra_main() # pylint: disable=no-value-for-parameter File “/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/main.py”, line 32, in decorated_main _run_hydra( File “/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/_internal/utils.py”, line 354, in _run_hydra run_and_report( File “/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/_internal/utils.py”, line 201, in run_and_report raise ex File “/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/_internal/utils.py”, line 198, in run_and_report return func() File “/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/_internal/utils.py”, line 355, in <lambda> lambda: hydra.multirun( File “/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/_internal/hydra.py”, line 136, in multirun return sweeper.sweep(arguments=task_overrides) File “/home/hello_automate_ai/miniconda3/lib/python3.10/site-packages/hydra/_internal/core_plugins/basic_sweeper.py”, line 140, in sweep sweep_dir.mkdir(parents=True, exist_ok=True) File “/home/hello_automate_ai/miniconda3/lib/python3.10/pathlib.py”, line 1179, in mkdir self.parent.mkdir(parents=True, exist_ok=True) File “/home/hello_automate_ai/miniconda3/lib/python3.10/pathlib.py”, line 1179, in mkdir self.parent.mkdir(parents=True, exist_ok=True) File “/home/hello_automate_ai/miniconda3/lib/python3.10/pathlib.py”, line 1179, in mkdir self.parent.mkdir(parents=True, exist_ok=True) File “/home/hello_automate_ai/miniconda3/lib/python3.10/pathlib.py”, line 1175, in mkdir self._accessor.mkdir(self, mode) PermissionError: [Errno 13] Permission denied: ‘/checkpoint’ Traceback (most recent call last): File “/home/hello_automate_ai/fairseqmmstest/fairseq/examples/mms/asr/infer/mms_infer.py”, line 52, in <module> process(args) File “/home/hello_automate_ai/fairseqmmstest/fairseq/examples/mms/asr/infer/mms_infer.py”, line 44, in process with open(tmpdir/“hypo.word”) as fr: FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmp0mcwde4n/hypo.word’
Getting same error. Also documentation to run sample is horrible.
@epk2112 Thanks for the notebook. I followed everything you did but the audio im using gives me this error
AssertionError: Sentences lengths should not exceed max_tokens=4000000Does MMS ASR support long audio files? the audio im using is 31 minutes. Here is the full errorHi, thanks for this discussion - I’ve learned a lot. This is the Dockerfile I created after a few hours trying to make it work:
I built the image with:
And run it with:
Yeah, that’s what I mean, if anything happens within the subprocess for any reason, folks are going to get the above mentioned error. Then they will likely google their way into this issue, which covers many of the possible ways it can fail. I was trying to be extra verbose for other folks to potentially help.
Thanks! You helped a lot, I eventually had to rewrite that whole block like so:
To even have the command execute and do something and not fail outright.
Hi all, If someone is still struggling to run the code, I tried to create a Python package to easily use the MMS project, instead of calling
subprocessand dealing withyamlfiles. `Hope it will be useful! 😃I got this error when i want to try ASR on google colab
Got the model to finally load and run, apparently windows doesn’t allow : in directory names and the above code adds
:devto the directory name.So if you pass
--lang turlike I did, it will try to create a directory named/tur:devinside the/checkpointwhich per @audiolion I also had to… change as/checkpointdoesn’t seem to do anything on windows.I think the full inference ran, as the process got stuck for a few minutes, the GPU went to 8GB (impressive) and after a while, I had 2 errors again.
the
hypo.worderror seems to be a “catch all” error that means… many things that could go wrong, hopefully the authors will clean it up?I’m currently staring at this error, and am pretty sure that’s due to me removing the
:from the dir nameSure here is the full log of mine
(base) hello_automate_ai@machinelearningnotebook:~/fairseqmmstest/fairseq$ python “examples/mms/asr/infer/mms_infer.py” --model “/home/hello_automate_ai/fairseqmmstest/mms1b_all.pt” --lang hin --audio “/home/hello_automate_ai/fairseqmmstest/audio.wav” preparing tmp manifest dir … loading model & running inference … Traceback (most recent call last): File “/home/hello_automate_ai/fairseqmmstest/fairseq/examples/speech_recognition/new/infer.py”, line 18, in <module> import editdistance ModuleNotFoundError: No module named ‘editdistance’ Traceback (most recent call last): File “/home/hello_automate_ai/fairseqmmstest/fairseq/examples/mms/asr/infer/mms_infer.py”, line 52, in <module> process(args) File “/home/hello_automate_ai/fairseqmmstest/fairseq/examples/mms/asr/infer/mms_infer.py”, line 44, in process with open(tmpdir/“hypo.word”) as fr: FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmp6u8grbxl/hypo.word’
you need to check what the error is, change your
mms_infer.pytoto see the error, for me it was I needed to pass
cpu=Truebecause I don’t have CUDA installed. I did this by modifying myinfer_common.ymlfile to have a new top level keycommonwith thecpu: truekey/val in it@audiolion We expect a 3-digit language code. See ‘Supported languages’ section in README file for each model. For example - use ‘eng’ for English.
@bekarys0504 , yes it is a right model it seems. Could you please submit an issue on the project repo so we can debug this issue further together ?
Ok for anyone who still have the
FileNotFoundError: [Errno 2] No such file or directoryfor thehypo.worderror and just want to test the inference:Its really what the error says. 😄 While inference the programming accesses the tmp folder and need to write some file. Including the hypo.word. As the error says: in line 44 of the mms_infer.py the it trys to open and write the
hypo.word.with open(tmpdir/"hypo.word") as fr:as you can see no righs are defined for theopenmethod are defined. so just give python the right to write and read the file.with open(tmpdir/"hypo.word", "w+") as fr:this should be all.you can see in the code
python should already created tthe files dev.tsv, dev.uid, dev.ltr and dev.wrd in the same tmp folder. If you want to check this, simply change the
tmpdir = Path(tmpdir)in to a static folder for instance in your user directory liketmpdir = Path("/home/myuser/path/to/my/project/test")and you will see that those file will bne created. including the
hypo.wordif you did the changes like I discribed before.now the the
examples/speech_recognition/new/infer.pywill be triggerd in line 40. and it might fail writing down the inference log file. like @v-yunbin discribed `FileNotFoundError: [Errno 2] No such file or directory: ‘/checkpoint/…/INFER/None’and its again just a problem with permissions to write some files. next to the
mms_infer.pyfile is aconfigfolder including ainfer_common.yamland there is the propertyso it trys to write in to the
checkpointfolder on root level. If you can not do that. simply change tthis folder my some folder in your user folder: for instanceso now the script will have access to those folders and whrite down the inferece log (infer.log) in to that folder which includes the result of the ASR.
@luisroque @fcecagno To accommodate longer sentence, you can change max_tokens=<new_value>. The calculation is 16000 sample per seconds * 30 minutes * 60 seconds per minutes = 28,800,000 max_tokens size. However, I would recommend you to split the long audio into shorter audio, because large max_tokens will cause GPU OOM or run much slower.
I had the same error with Google Colab and have investigated.
my error
As it turns out, it was crashing at the following location.
https://github.com/facebookresearch/fairseq/blob/af12c9c6407bbcf2bca0b2f1923cf78f3db8857c/fairseq/models/wav2vec/wav2vec2.py#L1052
Looking at the RAM status, I believe the crash was caused by lack of memory.
So I feel that perhaps increasing the memory will solve the problem.
I hope this helps you in your investigation.
Thanks @audiolion It wasn’t immediately clear that
mms_infer.pycalls the whole hydra thing via a command, as it obscures the errors that pop up there.Here’s the full output I’m getting (added a print out of the cmd command as well)
Getting pretty much the same, used the right 3 letter language code (while waiting on #5119 to be answered) and doesn’t seem to have an effect,
hypo.worderror is showing up@shsagnik
you need to install the modules that are used
@shsagnik
No module named 'editdistance'- You should install the missing module.Here we go: https://colab.research.google.com/drive/1jqREwuNUn0SrzcVjh90JSLleSVEcx1BY?usp=sharing simple 4 cell colab
I edited the script and it’s now working for me, with an Ubuntu 22.04 image, tested with both CUDA 11.8 and 12.1. Note that I added permissions for /checkpoint/${USERNAME}.
Dockerfile.mms:Building with:
Running with:
Guys if you are free google collab user, memory kills process. So use mms1b_fl102 model instead of mms1b_all
I will update if i have a fix
Hello, thank you for your response. I have indeed executed the code you mentioned.
After series of tries, i was able to get it to infer on Linux but it could probably work on Windows also. The hypo.word file missing error is due to exceptions thrown during
subprocess.run(cmd, shell=True, stdout=subprocess.DEVNULL,)so first i suggest you replace that line with the following:This will enable you see whats causing the error. Also provide the full paths of your model and audio files like this
python examples/mms/asr/infer/mms_infer.py --model "/home/hunter/Downloads/mms1b_all.pt" --lang eng --audio "/home/hunter/Downloads/audio.wav"It depends on how large your GPU size is, but try to split your audio, if possible, to avoid the OOM. Check this comment: https://github.com/facebookresearch/fairseq/issues/5117#issuecomment-1561787185
If you are using Linux you can try this. It runs with no problem for me.
30 min would be good enough, 1 hour would be even better.
I managed to run it on a Linux machine; thanks for the tips.
A few useful points:
You can find the Python script that I used to run ASR with MMS here with some more details here. I hope it helps!
I used your Dockerfile but it doesn’t read audio files:
@hrishioa
The
dataclasserror may be related to Python3.11(#5012). Switching back to Python3.9.16may resolve the problem.I can run the inference after installing all packages at here and fixing the checkpoint path at here.
Also, I find that
--modelshould specify the absolute path to the model file. A relative path like../model.ptmay cause an extra error.Same error.
SIGH, I am, it prints the command and that’s it.
However, when I go back and recreate that temp dir, and run the command manually myself I do seem to get errors.
Just for some reason not via the way you mentioned.
Had to install many packages on the way, here’s a partial list (in case it helps anyone)
Still getting nowhere. Running the subprocess command even with
check=Trueand printing the output returns status code 0 with no errors.Facing the exact same issue