STT: Bug: ValueError: Cannot feed value of shape (32,) for Tensor 'layer_6/bias/Initializer/zeros:0', which has shape '(29,)'

Hi all 😃

I am using commands which do work on a US english dataset, and I am now trying on a very small dataset (only 15 lines/wave files), which have Danish letters like æ, ø and å. I sync’ed with Github about a week ago.

When running this command: python3 ~/Python-3.7.6/STT/lm_optimizer.py --alphabet_config_path ~/Python-3.7.6/STT/data/alphabet.txt --scorer_path ~/kenlm.scorer --test_files ~/talefiler/train.csv ~/talefiler/dev.csv --checkpoint_dir ~/coqui-stt-1.4.0-checkpoint --n_trials 6 --n_hidden 256 --lm_alpha_max 5 --lm_beta_max 5

I get this error: ValueError: Cannot feed value of shape (32,) for Tensor ‘layer_6/bias/Initializer/zeros:0’, which has shape ‘(29,)’

Longer log here:

[I 2022-07-05 14:35:33,907] A new study created in memory with name: no-name-f467ce79-fc9c-48fb-bdc6-5cfcef882133
I Loading best validating checkpoint from /home/bruger/coqui-stt-1.4.0-checkpoint/best_dev-1017
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias
I Loading variable from checkpoint: cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
I Loading variable from checkpoint: global_step
I Loading variable from checkpoint: layer_1/bias
I Loading variable from checkpoint: layer_1/weights
I Loading variable from checkpoint: layer_2/bias
I Loading variable from checkpoint: layer_2/weights
I Loading variable from checkpoint: layer_3/bias
I Loading variable from checkpoint: layer_3/weights
I Loading variable from checkpoint: layer_5/bias
I Loading variable from checkpoint: layer_5/weights
I Loading variable from checkpoint: layer_6/bias
[W 2022-07-05 14:35:35,909] Trial 0 failed because of the following error: ValueError("Cannot feed value of shape (32,) for Tensor 'layer_6/bias/Initializer/zeros:0', which has shape '(29,)'")
Traceback (most recent call last):
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/lm_optimize.py", line 39, in objective
    current_samples = evaluate([test_file], create_model)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/evaluate.py", line 99, in evaluate
    load_graph_for_evaluation(session)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 233, in load_graph_for_evaluation
    _load_or_init_impl(session, methods, allow_drop_layers=False, silent=silent)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 171, in _load_or_init_impl
    silent=silent,
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 127, in _load_checkpoint
    load_cudnn=Config.load_cudnn,
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 91, in _load_checkpoint_impl
    v.load(ckpt.get_tensor(v.op.name), session=session)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1033, in load
    session.run(self.initializer, {self.initializer.inputs[1]: value})
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1156, in _run
    (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (32,) for Tensor 'layer_6/bias/Initializer/zeros:0', which has shape '(29,)'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/lm_optimize.py", line 97, in <module>
    main()
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/lm_optimize.py", line 86, in main
    results = compute_lm_optimization()
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/lm_optimize.py", line 59, in compute_lm_optimization
    study.optimize(objective, n_jobs=1, n_trials=Config.n_trials)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/optuna/study/study.py", line 409, in optimize
    show_progress_bar=show_progress_bar,
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 76, in _optimize
    progress_bar=progress_bar,
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 163, in _optimize_sequential
    trial = _run_trial(study, func, catch)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 264, in _run_trial
    raise func_err
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/lm_optimize.py", line 39, in objective
    current_samples = evaluate([test_file], create_model)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/evaluate.py", line 99, in evaluate
    load_graph_for_evaluation(session)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 233, in load_graph_for_evaluation
    _load_or_init_impl(session, methods, allow_drop_layers=False, silent=silent)
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 171, in _load_or_init_impl
    silent=silent,
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 127, in _load_checkpoint
    load_cudnn=Config.load_cudnn,
  File "/home/bruger/Python-3.7.6/STT/training/coqui_stt_training/util/checkpoints.py", line 91, in _load_checkpoint_impl
    v.load(ckpt.get_tensor(v.op.name), session=session)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1033, in load
    session.run(self.initializer, {self.initializer.inputs[1]: value})
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/bruger/Python-3.7.6/coqui-stt-train-venv/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1156, in _run
    (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (32,) for Tensor 'layer_6/bias/Initializer/zeros:0', which has shape '(29,)'

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 16

Most upvoted comments

HarikalarKutusu, I’ll do that. I have come around the bug by letting Coqui create an aphabet.txt on it’s own too, and then use that. On a small dataset, it doesn’t use all the letters - and using a full alphabet seems to crash Coqui. But working around it, I have been able to create working scorers + models, not even using transfer-train. This + using low n_hidden value got me down to 0.5% error-rate.

Thank you very much for your help, your video is very well made and I appreciate your help.

FrontierDK on Jul 16, 2022

HarikalarKutusu, you have made a very good video - thank you for sharing it 😃

Specifying different loading and saving folders kinda work, but I get this warning: WARNING: You specified different values for --load_checkpoint_dir and --save_checkpoint_dir, but you are running training and testing in a single invocation. The testing phase has been disable to prevent unexpected behavior of testing on the base checkpoint rather than the trained one. You should train and evaluate in two separate commands, specifying the correct --load_checkpoint_dir in both cases.

FrontierDK on Jul 7, 2022