hydra: [Bug] Submitit Handling of ListConfigs

๐Ÿ› Bug

Description

Iโ€™m currently using your master branch (love the direction this is going) and have found a bug that may exist for your released version as well. If I add the following parameter to my SlurmLauncher config:

setup:
  - export MASTER_ADDR=$(hostname -s)
  - export MASTER_PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')

I get the following error:

Traceback (most recent call last):
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/hydra/_internal/utils.py", line 196, in run_and_report
    return func()
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/hydra/_internal/utils.py", line 356, in <lambda>
    overrides=args.overrides,
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 137, in multirun
    return sweeper.sweep(arguments=task_overrides)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/hydra/_internal/core_plugins/basic_sweeper.py", line 154, in sweep
    results = self.launcher.launch(batch, initial_job_idx=initial_job_idx)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/hydra_plugins/hydra_submitit_launcher/submitit_launcher.py", line 122, in launch
    executor.update_parameters(**params)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/submitit/core/core.py", line 638, in update_parameters
    self._internal_update_parameters(**kwargs)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/submitit/auto/auto.py", line 197, in _internal_update_parameters
    self._executor._internal_update_parameters(**parameters)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/submitit/slurm/slurm.py", line 307, in _internal_update_parameters
    _make_sbatch_string(command="nothing to do", folder=self.folder, **kwargs)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/submitit/slurm/slurm.py", line 490, in _make_sbatch_string
    lines += ["", "# setup"] + setup
TypeError: can only concatenate list (not "ListConfig") to list

I tried adding _convert_: all to the config, but get the following error:

Traceback (most recent call last):
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/hydra/utils.py", line 99, in instantiate
    convert = _pop_convert_mode(final_kwargs)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/hydra/_internal/utils.py", line 631, in _pop_convert_mode
    convert = d.pop("_convert_")
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 453, in pop
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/omegaconf/base.py", line 113, in _format_and_raise
    type_override=type_override,
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/omegaconf/_utils.py", line 678, in format_and_raise
    _raise(ex, cause)
  File "/home/gridsan/JU17693/.conda/envs/raiden-release/lib/python3.7/site-packages/omegaconf/_utils.py", line 588, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ReadonlyConfigError: Cannot pop from read-only node
        full_key: hydra._convert_
        reference_type=Optional[Dict[Union[str, Enum], Any]]
        object_type=dict

Checklist

  • I checked on the latest version of Hydra
  • I created a minimal repro

System information

  • Hydra Version : latest master
  • Python version : 3.7
  • Virtual environment type and version : conda
  • Operating system : Linux

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 17 (8 by maintainers)

Most upvoted comments

@jrapin, any reason why setup should not be a member of SlurmQueueConf in the plugin?

Just that Iโ€™ve never used it so didnโ€™t consider it, but if itโ€™s useful, no reason not to