hydra: [Bug] Cannot use plugin's conf dataclasses in structured config
🐛 Bug
Description
Hi,
First of all, many thanks for that wonderful tool you’ve made. It is, for sure, extremely helpful to have it when production environment is necessary. I have met an error/bug when I wanted to use the OptunaSweeperConf using the Python API (instead of using the traditional YAML files).
Checklist
- I checked on the latest version of Hydra
- I created a minimal repro (See this for tips).
To reproduce
** Minimal Code/Config snippet to reproduce **
# Standard libraries
from dataclasses import dataclass
# Third-party libraries
import hydra
from hydra.conf import HydraConf
from hydra.core.config_store import ConfigStore
from hydra_plugins.hydra_optuna_sweeper.config import OptunaSweeperConf
from omegaconf import DictConfig
@dataclass
class ExampleConfig():
hydra: HydraConf = HydraConf(sweeper=OptunaSweeperConf())
ConfigStore.instance().store(name='config', node=ExampleConfig)
@hydra.main(config_path=None, config_name='config')
def main(config: DictConfig) -> None:
"""Main function."""
print(config)
if __name__ == '__main__':
main()
And the stack trace with the returned error:
** Stack trace/error message **
In 'config': Validation error while composing config:
Merge error: OptunaSweeperConf is not a subclass of BasicSweeperConf. value: {'_target_': 'hydra_plugins.hydra_optuna_sweeper.optuna_sweeper.OptunaSweeper', 'defaults': [{'sampler': 'tpe'}], 'sampler': '???', 'direction': <Direction.minimize: 1>, 'storage': None, 'study_name': None, 'n_trials': 20, 'n_jobs': 2, 'search_space': {}}
full_key:
object_type=dict
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Expected Behavior
I would like to override the hydra config group, directly into the structure config. To do so, I import the HydraConf from hydra and install the hydra-optuna-sweeper plugin. The error seems to indicate that the OptunaSweeperConf must inherit from the BasicSweeperConf structured config (current implementation: https://github.com/facebookresearch/hydra/blob/v1.1.1/plugins/hydra_optuna_sweeper/hydra_plugins/hydra_optuna_sweeper/config.py#L133). The idea is to be able to override hydra config group directly from a python script. Does that make sense?
System information
- Hydra Version : 1.1.1
- Hydra Optuna Sweeper Version : 1.1.1
- Python version : 3.8.11
- Virtual environment type and version : MiniConda environment
- Operating system : LinuxMint
Additional context
Add any other context about the problem here.
About this issue
- Original URL
- State: open
- Created 3 years ago
- Comments: 22 (7 by maintainers)
@Jasha10 you are right! 🤣 i updated the previous comment.
Ok - I took another look at this and here is what is happening: the tl;dr is the plugins are being imported twice and as a result OmegaConf fails the validation here
OptunaSweeperConf is not a subclass of OptunaSweeperConf. In debug mode, comparing the dest_obj_type and src_obj_type, i can see that although they are the same class, they have different ids (meaning they are different types due to the double imports)I also found a so question that describes a similar issue here
Worth trying to figure out the root cause.
Hi @jieru-hu, Sorry for the explanation, I think I am not clear enough. I am perfectly aware of how to configure a sweeper using YAML files (this is well documented) or directly through the command line. My question was about configuring a Sweeper directly using the Python API (structured config)? But your answers suggest that it is not possible or not proposed by hydra.
To put it in some context, I’m using Hydra to run Deep Learning trainings using PyTorch-Lightning like that:
I really like to use the Python API because it offers the benefits of using an IDE (like VSCode) to easily manipulate structured configurations with auto-completion. And I would like to be happy to override HydraConf directly in that file (having a single file to centralize all the config used for training). I managed to override the run dir like so, with adding the hydra config group:
But if I want to do the same thing with a sweeper (like Optuna), it fails (see the error at the beginning of the thread). I understand that Hydra relies essentially on manipulating and composing YAML files but I also see an interest of manipulating the structured config (the schemas) directly through a Python script. Maybe I am wrong 😃 .
PS: there was an attempt, in https://github.com/pytorch/hydra-torch/blob/main/examples/mnist_00.py to use directly structured configs through a script. And I think we can go a step further with the example above. What do you think?
Hi @Jasha10,
I have made some tests using your PR with the following example (simply implementing the sphere example but using the python API:
Running with following command
python sphere.py --multirun, I get this output:Which seems to be fine for me. 😃
Nice detective work!!
Here is a minimal reproducer for the issue based on the comment above:
I have an idea for a diff to solve the issue.
By doing the import with
importlib.import_moduleinstead ofpkgutil, we get the same module as is loaded by theimportstatement at the top of the file.Hello 😃 Do you have any update on that topic? Or need some help to further investigate on that?
sorry @mayeroa for getting back to you late, thanks for further explaining your use case.
like @omry suspected earlier, it does seem to be related to how the two classes are imported, in particular, the application fails at merging the configs here, although the two Conf classes have the same type, the
issubclassreturn False.I tried to move the
ConfigStorecall to the__init__.pylike the followingwhich seems to resolve the issue, and the config override worked running @Jasha10’s example above
I’m not sure if this is the best way to fix this though, @omry thoughts?
This is probably related to how plugins are loaded discovered and loaded. There are two different OptunaSweeperConf classes loaded.
hi @mayeroa pls try using the defaults list instead. something like the following should work
Run the application, you should see optuna’s config
Also the
cfgobject you get viahydra.maindoes not contain Hydra’s config, to access the configs, try something like the following