NeMo: Conformer-Transducer Small and Medium models give ERROR: omegaconf.errors.ConfigAttributeError: Key 'att_context_size' is not in struct

Describe the bug I am getting an error when running inference on a test dataset using the Conformer-Transducer Small and Medium models downloaded from NGC. I do not get receive this issue when using the Large, XLarge and XXLarge models. However, since I am interested in running inference on embedded devices, I would like to use the Small and Medium models more than the Large ones.

The error is as follows:

Traceback (most recent call last):
  File "transcribe_speech.py", line 327, in <module>
    main()  # noqa pylint: disable=no-value-for-parameter
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/config/hydra_runner.py", line 105, in wrapper
    _run_hydra(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 453, in <lambda>
    lambda: hydra.run(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "transcribe_speech.py", line 200, in main
    asr_model, model_name = setup_model(cfg, map_location)
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/parts/utils/transcribe_utils.py", line 191, in setup_model
    asr_model.change_attention_model(
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/parts/mixins/mixins.py", line 426, in change_attention_model
    self.encoder.change_attention_model(self_attention_model, att_context_size, update_config, self.device)
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/modules/conformer_encoder.py", line 841, in change_attention_model
    att_context_size = self._cfg.att_context_size
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/_utils.py", line 820, in format_and_raise
    _raise(ex, cause)
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/_utils.py", line 798, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/_utils.py", line 900, in format_and_raise
    _raise(ex, cause)
  File "/usr/local/lib/python3.8/dist-packages/omegaconf/_utils.py", line 798, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
omegaconf.errors.ConfigAttributeError: Key 'att_context_size' is not in struct
    full_key: att_context_size
    object_type=dict

Steps/Code to reproduce bug The main script I am trying to run is https://github.com/NVIDIA/NeMo/blob/main/examples/asr/transcribe_speech.py with the following command line call:

OC_CAUSE=1 HYDRA_FULL_ERROR=1 python3 transcribe_speech.py model_path=/workspace/nemo/non-finetuned/models/stt_en_conformer_transducer_small.nemo dataset_manifest=/workspace/datasets/myst_w2v2_asr/test_manifest.json output_filename=/workspace/nemo/non-finetuned/results/stt_en_conformer_transducer_xxlarge_myst_test_predictions.json batch_size=8 cuda=1 amp=True

Expected behavior

Expected to receive the following successful output, just like when using Conformer-Transducer Large models:

[NeMo I 2023-05-09 11:12:08 transcribe_utils:235]
    Transcribing 2566 files...

[NeMo I 2023-05-09 11:12:08 transcribe_speech:251] AMP enabled!

Transcribing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 321/321 [02:39<00:00,  2.01it/s]
[NeMo I 2023-05-09 11:14:48 transcribe_speech:305] Finished transcribing 2566 files !

Environment overview (please complete the following information)

• Environment location: Docker
• Method of NeMo install: docker pull nvcr.io/nvidia/nemo:23.02

Environment details

I used the latest NeMo docker image as my environment.

Additional context

GPU model: Nvidia A6000