NeMo: NeMo installation fails due to outdated abandoned dependency on youtokentome

Describe the bug

NeMo installation fails with Python >=3.9 because youtokentome does not properly specify it’s build dependency on cython. See this issue in their repo.. There is also a PR on that repo to fix that but the author seems to have abandoned the project, its last release was in 2020.

Steps/Code to reproduce bug

Trying to install nemo_toolkit using Python 3.9 or above.

Expected behavior

Installation should succeed.

Environment overview (please complete the following information)

Environment location: GCP
Method of NeMo install:poetry add nemo_toolkit[asr]

Environment details

If NVIDIA docker image is used you don’t need to specify these. Otherwise, please provide:

Debian
PyTorch 1.13.1 and >2.0
Python 3.9, 3.10, 3.11

Additional context

This is particularly annoying given that the whole package depends on youtokentome (see requirements_common.txt). So even if users like me only care about the ASR part, youtokentome must be installed.

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (2 by maintainers)

Most upvoted comments

This issue is very real, because even though there is a workaround when installing individual libraries using pip or poetry, trying to install from requirements.txt or from poetry.lock fails.

The easiest workaround seems to be to merge the PR @svandiekendialpad mentioned in a fork of the original repo, and then install that project before installing nemo-toolkit.

I have taken this approach, creating the fork @https://github.com/LahiLuk/YouTokenToMe. Leaving it here in case it can help someone as a workaround.

LahiLuk on Aug 17, 2023

Okay, these steps actually worked for me! 🎉

pip install pybind11
pip install wheel setuptools pip --upgrade
pip install fasttext
pip install "nemo_toolkit[all]"

askiefer on Mar 3, 2024

Now I get, RuntimeError: pybind11 install failed. Wow, never ends.

oerdem19 on Mar 1, 2024

We’re basically removing yttm entirely, in #8322

titu1994 on Feb 8, 2024

Hi @svandiekendialpad, sorry for the late reply. If this is still a problem, could you please open an issue in https://github.com/LahiLuk/YouTokenToMe/issues? Then I can investigate further. Thanks!

LahiLuk on Feb 1, 2024