STT: Bug: lm_optimize throws error

Hi

I’ve tried it a couple of different ways and lm_optimize() always comes back that lm_opt is not defined, as below.

I tried python -m coqui_stt_training.util.lm_optimize as well with the same error.

mike h

python lm_optimizer.py --test_files /mhdata/cvnew/clips/tst.csv --checkpoint_dir fordata/mhwelsh/welsh/checkpoints --n_hidden 200 --scorer_path fordata/newwelsh.scorer
Using the top level lm_optimizer.py script is deprecated and will be removed in a future release. Instead use: python -m coqui_stt_training.util.lm_optimize
2022-04-24 22:22:02.584403: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2022-04-24 22:22:04.263143: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2299965000 Hz
2022-04-24 22:22:04.265496: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x53a4130 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-04-24 22:22:04.265560: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2022-04-24 22:22:04.271188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-04-24 22:22:05.976601: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-24 22:22:05.976833: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x58bdfd0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2022-04-24 22:22:05.976853: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA GeForce RTX 2080 Super with Max-Q Design, Compute Capability 7.5
2022-04-24 22:22:05.977046: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-24 22:22:05.977177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1666] Found device 0 with properties:
name: NVIDIA GeForce RTX 2080 Super with Max-Q Design major: 7 minor: 5 memoryClockRate(GHz): 1.08
pciBusID: 0000:01:00.0
2022-04-24 22:22:05.977214: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-24 22:22:05.988396: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-04-24 22:22:06.017602: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-04-24 22:22:06.018920: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-04-24 22:22:06.022358: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2022-04-24 22:22:06.035457: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2022-04-24 22:22:06.035625: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2022-04-24 22:22:06.035725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-24 22:22:06.035919: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-24 22:22:06.036020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1794] Adding visible gpu devices: 0
2022-04-24 22:22:06.036259: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-04-24 22:22:07.039992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-24 22:22:07.040027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212]      0
2022-04-24 22:22:07.040034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0:   N
2022-04-24 22:22:07.040481: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-24 22:22:07.040668: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1082] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-24 22:22:07.040809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/device:GPU:0 with 6660 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Super with Max-Q Design, pci bus id: 0000:01:00.0, compute capability: 7.5)
I --alphabet_config_path not specified, but found an alphabet file alongside specified checkpoint (fordata/mhwelsh/welsh/checkpoints/alphabet.txt). Will use this alphabet file for this run.
Traceback (most recent call last):
  File "lm_optimizer.py", line 16, in <module>
    lm_optimize.main()
  File "/code/training/coqui_stt_training/util/lm_optimize.py", line 86, in main
    results = lm_opt.compute_lm_optimization()
NameError: name 'lm_opt' is not defined

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (1 by maintainers)

Most upvoted comments

Using the main solved this for me. Thanks @wasertech.

+1
SuperKogito on Jul 25, 2022

This still occurs in the 1.4.0a1 docker image

@sjpritchard Try with main instead of latest.

$ docker pull ghcr.io/coqui-ai/stt-train:main
$ docker run -it ghcr.io/coqui-ai/stt-train:main
+1
wasertech on Jun 8, 2022

I can confirm that the bug is now gone 😃

+1
FrontierDK on Apr 26, 2022

So what do we do with the alpha and beta produced by lm_optimizer.py? My best guess at this point is that we recreate the scorer?

@hammondm , not exactly. That was also my first impression thou…

  1. You have the acoustic model and scorer.
  2. You might like to run a test (.evaluate) with this non-optimized scorer (default alpha/beta)
  3. You run lm_optimize to find better values for alpha & beta, it is a random walk, the more steps, the better. It is a costly process. This will result in best alpha/beta values.
  4. You may run the test again with these new parameters to see if they are better.
  5. You use these values in the inference engine. The tflite acoustic model and your scorer will be the same. Use lm_alpha, lm_beta parameters…

Bülent Özden

+1
HarikalarKutusu on Apr 25, 2022

@FrontierDK wait for my branch to merge with main 😃

I’ll go to bed now (close to midnight here) and try a new fresh install tomorrow. Thank you for fixing 😃

+1
FrontierDK on Apr 25, 2022

I’m guessing you are using version 1.2.0. You can either upgrade to v1.3.0 or fix the import path manually. Replace

from coqui_stt_training.util.lm import optimize as lm_opt

With

from coqui_stt_training.util import lm_optimize as lm_opt

inside lm_optimizer.py. See #2116 and #2148 for more context.

+1
wasertech on Apr 25, 2022
Read more comments on GitHub
← STT: Bug: Batches are not evaluated in parallel
STT: Bug: ValueError: Cannot feed value of shape (32,) for Tensor 'layer_6/bias/Initializer/zeros:0', which has shape '(29,)' →